The search problem posed by large heterogeneous data sets in litigation

Abstract

Lawyers and their large institutional clients increasingly face the enormous problem of how to efficiently and efficaciously conduct searches for relevant documents in large heterogeneous electronic data sets, for the purpose of responding to litigation demands. Past research indicates that lawyers greatly overestimate their true rate of recall in civil discovery. The unprecedented size, scale, and complexity of electronically stored data now potentially subject to routine capture in litigation, for purpose of preservation, access, and review, presents information retrieval researchers with a series of important challenges to overcome. This paper describes the current context of e-discovery and discusses the potential for IR and AI research to address the challenges of conducting e-discovery. The TREC Legal Track is presented as a forum for the evaluation of e-discovery research and one new evaluation measure, elusion, is described, which has potential for addressing problems of measuring recall.

A history of AI and Law in 50 papers: 25 years of the international conference on AI and Law

“…The search problem posed by large heterogeneous data sets in litigation: possible future approaches to research [25]. Commentary by Dave Lewis 25 In November 2005, I had my first conversation with Jason Baron and was introduced to the fascinating world of electronic discovery (e-discovery), a topic which has since come to be the focus of both my consulting and my research.…”

“…Commentary by Dave Lewis 25 In November 2005, I had my first conversation with Jason Baron and was introduced to the fascinating world of electronic discovery (e-discovery), a topic which has since come to be the focus of both my consulting and my research. For many researchers in AI and Law, however, the Baron and Thompson paper at ICAIL 2007 [25] was the first flag of warning and opportunity in advance of the tsunami that has become e-discovery.…”

“…Most importantly, [25] conveyed the central role that random sampling and statistical evaluation would play in e-discovery. In 2012, this is beyond obvious.…”

Emerging AI & Law approaches to automating analysis and retrieval of electronically stored information in discovery proceedings

“…As materials for the competition, experts associated with the Sedona Conference (http://www.thesedonaconference.org/)-a continuing forum of jurists, attorneys and technologists addressing issues of complex litigation-have created nine fictitious but realistic legal complaints, and 53 retrieval topics cast as document requests each with a corresponding Boolean query created through a realistic process of negotiation (Baron and Thompson 2007;Oard et al 2010, this issue). For each topic, the National Institute of Standards and Technology generated judgment pools for relevance assessment based on searches of an expert manual searcher and the participating teams and manual assessments by 35 volunteer assessors (Baron and Thompson 2007;Oard et al 2010, this issue). The TREC Legal Track competition has also made public another similarly prepared repository of discovery requests and materials in connection with email documents generated by the Enron litigation (Oard et al 2010, this issue).…”

Making sense of archived e‐mail: Exploring the Enron collection with NetLens

“…A problem known as e‐discovery in the context of civil litigation has emerged as a driving application for effective and efficient search in large e‐mail collections (and many other types of digital data). As digital records displace physical records of individual, corporate, and government activity, enormous collections are being amassed that exceed what the manual review techniques of an earlier era can accommodate (Baron & Thompson, 2007). Development of search technology tailored to the requirements of this task is being investigated by participants in the Text Retrieval Conference's legal track (Oard, Hedin, Tomlinson, & Baron, 2008).…”