A recent decision by D.C. District Judge Colleen Kollar-Kotelly from In Re Domestic Airline Travel Antitrust Litigation provided the latest insight into how courts are dealing with the emergence of Technology Assisted Review (TAR) in discovery. In this case, Defendant United Airlines “produced more than 3.5 million [core] documents” with “approximately 17%, or 600,000 documents… responsive to Plaintiffs’ requests.” With this volume of non-responsive documents unanticipated, Plaintiffs motioned for an Extension of Fact Discovery Deadlines by six months to provide their attorneys time to review the received productions. The court granted the extension over Defendants’ objections.

Background

This case involves a multidistrict class action where Plaintiffs, airline passengers/customers, sued a group of commercial US airlines alleging a coordinated effort to reduce seat capacity and fix ticket prices was an unlawful restraint of trade. The court had “set a strict schedule for discovery and exhorted the parties to comply with the deadlines.” Two of the airlines reached settlements, and the remaining defendants, Delta and United, were scheduled for completion of core document production by April 30, 2018.

Because of the volume of potentially relevant material, United agreed with Plaintiffs that it would use Technology Assisted Review (“TAR”) to identify a population of documents for production. The parties agreed to a TAR protocol in which “United will set a minimum estimated recall rate of 75% but will endeavor to achieve a higher estimated recall rate if that rate may be obtained with a reasonable level of precision through reasonable additional training effort, taking into account the concept of proportionality and the deadline for substantial completion of document production.” To estimate recall and precision, the parties agreed to use both a Control Set, or a review of a random sample of documents conducted at the beginning of TAR training review, as well as a separate validation sample taken after the attorneys believed that TAR training was sufficient for production.

On April 27, 2018, just one business day before the production deadline, United notified Plaintiffs that their TAR training was complete and estimated their production would contain 85% of all responsive documents with a 58% precision rate. Stated differently, the forthcoming production by these estimates should contain at least one responsive document for every non-responsive document captured by TAR. On April 30, 2018, United made their production of 3.5 million documents.

Within a month following the production, Plaintiffs noticed that a separate Validation sample told a much different story from the Control Set estimates provided by United. The Validation sample indicated the production actually contained over 97% of all responsive documents, but the precision estimate was only 16.7%. This means that Plaintiffs would need to sort through at least four non-responsive documents for every one responsive document in the 3.5 million document production. It then took United attorneys another month to confirm the error of their original estimates. Following the acknowledgment, Plaintiffs’ submitted their motion to extend.

The Court analyzed the facts and determined Plaintiffs had shown the legal requirement of good cause for it to exercise its discretion to extend fact discovery. In its analysis, the court considered that Plaintiffs had worked diligently to develop a workable scheduling order, and that their compliance was hindered by “matters that were unforeseen,” i.e. “a much larger number of non-responsive documents than was anticipated” in the received production. The court additionally considered the relative prejudice of a six-month delay to each of the parties. While Plaintiffs needed time to review the production to engage in fact discovery with proper information, Defendants’ did not articulate any prejudice from delay other than the inconveniences of rescheduling.

Takeaways

TAR can save litigants millions of dollars in matters with astronomical data volumes such as this, but the outcome here provides a useful lesson that as easy as TAR may seem to use, there is no easy button for matters with this type of data volume and complexity of scale. Why did United’s attorneys believe their production contained over three times as many responsive documents as it did? A review of the correspondence submitted as exhibits in the attorney memorandums reveals they misread statistics presented by their TAR software. For matters of this magnitude, experts who work with this software daily need to be consulted early in the process, not after productions have already gone out the door and complaints have been received.

We hope the outcome here does not turn discovery practitioners away from TAR. The mistakes here are easily avoidable with proper analysis. While the court does not discuss hypotheticals, much of its analysis focused on the fact that the volume of non-responsive documents in the production was an unforeseen challenge presented to Plaintiffs. It may be that with better communication and discussion in advance of the production, the parties could have negotiated a production recall/precision that would have avoided the extension. TAR can also be used to simply prioritize review in matters with more manageable data volumes using what’s commonly known as a TAR 2.0 workflow. For example, CDS offers Brainspace Continuous Multi-Modal Learning or Relativity Active Learning which can be useful and require little expert intervention.

CDS’s Advisory Services team regularly consults on the use of TAR in eDiscovery. Contact us today and learn how TAR may be able to streamline your next matter.