White Glove Service for San Diego’s Hospitality Industry: Driven by Passionate People, Stokes Wagner Provides 5-Star Legal Services to Those in the... People Skills, Digital Leadership, Empowered Clients: Jackson & Wilson Showcase the Human Side of Law and Business withCutting-Edge... Attorney Safety: Interview With Stephen Kelson: Earlier this year, I shared statistics about threats of violence made against attorneys.... What Makes Your Firm Stand Out?: Tell us if you’ve met this guy before: John Smith is a personal injury attorney who... Compartmentalize and Integrate: Five Ways to Handle it ‘All’: I am a lawyer and a parent. I also try to have a social life, volunteer, take care of my... Tips for the Well- Dressed (Male) Lawyer: Every new parent knows there are far more clothing options for baby girls than baby boys.... Community News – September 2017: Attorney Antoinette Middleton, founder of the Law Offices of Antoinette Middleton, was... Tried, Tested & Triumphant: Khashayar Law Group Secures Position as One of Southern California’s Top... 7 Ways to Make Billing More Effective: Lawyers may practice in different areas of the law and serve a wide range of clients, but... Protecting Your Firm from Ransomware: Ransomware attacks are affecting every type of business in America, and law firms are no...
Executive Presentations-468x60-1

7 Practical Suggestions to Get the Most out of Technology-Assisted Review

Virus and hacking concept

With the recent high profile use of technology-assisted review in the Da Silva Moore and Kleen cases, and with positive judicial opinions coming from Judge Peck and others, many legal teams are considering using technology-assisted review for the first time. Without expert guidance, newcomers frequently struggle with the tools and processes of technology-assisted reviews. Here are some real-world guidelines for achieving the best results.

The basic underlying technologies have been around for decades. Machine learning techniques are used in a wide variety of industrial applications, including speech recognition, facial expression categorization, handwriting recognition, and computational biology, to name a few. Applied to the problem of large-scale legal document review and production, technology-assisted review enables a small number of expert human reviewers to train a computerized system to recognize the subtle differences between responsive and non-responsive documents based on the review of a small but representative sample of the document collection.

With explosive growth in electronically stored information, a system that magnifies the efforts of a single reviewer (such as assisted review) is seen as an important step towards increasing electronic discovery quality while simultaneously reducing cost and time.

To take best advantage of this new technology, here are some practical tips based on our experience in supporting dozens of real-life cases.

1. Use subject matter experts to train the system. Technology-assisted review magnifies the decisions made by the reviewers training the system. For consistency, the same set of expert reviewers should train the system from start to finish. This is something that can be delegated to outsourced review management in some cases. For example, a case involving an intellectual property dispute related to biotechnology may require experts in that field to determine which documents are relevant.

2. Use a review manager that understands technology-assisted review. For large cases, law firms and corporations often rely on outsourced review management. The review management team is often in the best position to use assisted review most effectively. Assisted review can be used in different ways depending on the specific needs of a case. These include prioritizing document review, using flexible staffing models to distribute review, bulk classification of non-relevant documents, and early case assessment, to name a few. Before starting, plan out how the technology and associated workflow fit into the electronic discovery plan for a given project. Having a review manager and a separate assisted review team can reduce the possible benefits if these teams don’t work closely together. Sidestep the issue by using a review manager well-versed in the assisted review process employed.

3. Learn the terms. There are several words frequently used in technology-assisted review that have a distinctive meaning. Richness, recall, precision, and f-measure are four key terms to know. Richness is the overall percentage of the document population that is responsive to an issue. Recall is the percent of the responsive documents identified by the process and is a measure of completeness. Precision is the percent of documents identified by the process as responsive that are actually responsive and is a measure of quality. F-measure is the weighted average of precision and recall. There are many more to learn, and knowing the vocabulary helps.

4. Apply technology-assisted review before culling with search terms. Using Boolean search terms to cull down the population ahead of assisted review can reduce the overall recall potential. A limitation of Boolean search is that you often find only what you know to look for. Using Boolean search requires the case team to know exactly what language was used in the target documents ahead of time. The strength of technology-assisted review is that it can uncover pockets of relevant content that were not suspected. Most assisted review processes sample across the document universe.

5. Use blind sampling to measure progress and avoid reviewer bias. Many technology-assisted review processes build in sampling to measure progress and provide quality control throughout review. When reviewers are aware that a document is marked responsive or not responsive, it can influence their decision. Representative sampling is a well-established method of determining the characteristics of a large population of documents.

6. Focus on one or two issues at most. Assisted review benefits are highest when ranking documents against the primary issues at the heart of a matter. There are generally dozens of sub-issues. Subdividing responsive documents into the many subcategories is best left to second pass reviewers with explicit review instructions and training.

7. Use second pass linear review. Use a second pass review to identify additional non-responsive documents as well as to identify documents to be withheld for privilege.

Technology-assisted review promises to reduce cost and improve quality compared to other available methods. With the exploding growth of electronically stored information, these new tools are needed. The legal teams that master them first will have a great advantage.

Jon Lavinder

Jon Lavinder, director of technology-assisted review at DTI.

More Posts

Digg This
Reddit This
Stumble Now!
Buzz This
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Kick It on DotNetKicks.com
Shout it
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)
PDF24    Send article as PDF   

Filed Under: Technology

About the Author: Jon Lavinder, director of technology-assisted review at DTI.

RSSComments (0)

Trackback URL

Leave a Reply

  • Polls
    Sorry, there are no polls available at the moment.