Evaluation of information retrieval for E-discovery

Oard, Douglas W.; Baron, Jason R.; Hedin, Bruce; Lewis, David; Tomlinson, Stephen

doi:10.1007/s10506-010-9093-9

Cited by 54 publications

(29 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Discussionsupporting

confidence: 91%

“…For instance, participants submitting the recall term 'EOL' produced a retrieval result that was significantly higher in recall (relevant documents) than the other participants in their group who did not submit that term. This is consistent with other research in the field supporting the notion that retrieval is highly sensitive to choice of search terms and can often be effected by the context of the subject matter being searched [3], [4].The significant results produced for reduction in nonrelevant documents due to the use of elimination terms are encouraging given that the time and cost associated with manual, human review can be reduced if there are significantly fewer non-relevant documents in the retrieval result [4]. In other words, a smaller total set to review, and fewer false positives to have to wade through.…”

supporting

confidence: 91%

“…This is an interesting task, given that the initial goal is not relevance, but learning the context of what might be relevant. In some cases, litigators are accused of "fishing" for information [3]. Fishing is an interesting metaphor for the volume problem, because if one pictures casting a large net to catch relevant fish, this can lead to the over inclusion in the IR result (too many non-relevant documents in the retrieval set).…”

mentioning

confidence: 99%

See 2 more Smart Citations

Using Recall and Elimination Terms in Separate Runs for High Volume Document Sorting

Hyman¹,

Sincich²,

Will³

et al. 2016

IJMLC

View full text Add to dashboard Cite

Abstract-This paper reports on a study undertaken to explore the problem of volume in searching large scale digital collections. An experiment is conducted using elimination terms as a method to reduce the number of non-relevant documents in the information retrieval (IR) result. The goal is to provide insight into how elimination terms can be used as a sorting method to reduce volume. The results of the experiment demonstrate that modifying the search structure with an elimination term component can significantly reduce the number of non-relevant documents in the retrieval set to address the problem of high volume in electronic document sorting and searching tasks.Index Terms-Information retrieval, knowledge discovery, search methods, document sorting. I. INTRODUCTIONVolume is one of the "three Vs" associated with large scale digital information (volume, variety and veracity), [1]. In this paper we focus on the volume problem by asking the question: How can we apply sorting methods to reduce the volume of electronic documents for sorting and searching tasks?A current trending problem is that, given the large volumes of information contained in electronic stores, search tools need to support the retrieval of relevant documents from large collections without producing too many non-relevant documents [2]- [4].A relevant document is defined as a document that meets the user's needs (see VanRijsbergen, 1979). A non-relevant document is defined as a document that does not meet the user's needs. To address the problem of high volume in digital collections, an effective method for sorting documents is helpful to return relevant documents, and not return non-relevant documents, in the IR result [5].An additional concern is about retrieving documents that are unauthorized, such as privileged or private documents (treated as non-relevant for sorting purposes). Also, the high cost associated with manual review of documents, both paper and electronic, increases the priority of using an automated method for sorting relevant and non-relevant documents, and reducing the number of documents in the retrieval set that must ultimately be settled by human Manuscript received October 16, 2015; revised February 4, 2016. Harvey Hyman is with New College of Florida, 5800 Bay Shore Road, Sarasota, Florida 34243, USA (e-mail: hhyman@NCF.edu).Terry Sincich and Rick Will are with University of South Florida, 4202 E. Fowler Avenue, Tampa, Florida 33620, USA (e-mail: tsincich@USF.edu, rwill@usf.edu).Warren Fridy III is with H2 & WF3 Research, LLC 701 S. Howard Avenue, Tampa, Florida 33606, USA (e-mail: warren@h2wf3.com).review-the most expensive part of the process.Precision, which we define in this paper as the percentage of relevant documents returned in the search result, is the measure used in this reported experiment to determine efficiency in a search result. Our goal here is that, if precision can be improved, then a smaller collection, with a greater percentage of relevant documents, can be produced by the automated system for huma...

show abstract

Section: Discussionsupporting

confidence: 91%

supporting

confidence: 91%

mentioning

confidence: 99%

See 1 more Smart Citation

Using Recall and Elimination Terms in Separate Runs for High Volume Document Sorting

Hyman¹,

Sincich²,

Will³

et al. 2016

IJMLC

View full text Add to dashboard Cite

show abstract

“…This may be the case, e.g., when classification is to be applied to a recall-oriented task (such as e-discovery [16]), in which case values β > 1 are appropriate. In these cases our utility-theoretic method can be used once the appropriate instance of F β is used in Equations 5 and 6 in place of F1.…”

Section: Discussionmentioning

confidence: 99%

A utility-theoretic ranking method for semi-automated text classification

Berardi

Esuli

Sebastiani

2012

Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

In Semi-Automated Text Classification (SATC) an automatic classifierΦ labels a set of unlabelled documents D, following which a human annotator inspects (and corrects when appropriate) the labels attributed byΦ to a subset D of D, with the aim of improving the overall quality of the labelling.An automated system can support this process by ranking the automatically labelled documents in a way that maximizes the expected increase in effectiveness that derives from inspecting D . An obvious strategy is to rank D so that the documents thatΦ has classified with the lowest confidence are top-ranked. In this work we show that this strategy is suboptimal. We develop a new utility-theoretic ranking method based on the notion of inspection gain, defined as the improvement in classification effectiveness that would derive by inspecting and correcting a given automatically labelled document. We also propose a new effectiveness measure for SATC-oriented ranking methods, based on the expected reduction in classification error brought about by partially inspecting a list generated by a given ranking method. We report the results of experiments showing that, with respect to the baseline method above, and according to the proposed measure, our ranking method can achieve substantially higher expected reductions in classification error.

show abstract

“…One recall-oriented task is e-discovery [62,63], in which the goal is to retrieve documents from large collections that are relevant to a request for production in civil litigation. The documents are usually reviewed by a team of experts, so examining an entire collection is an expensive manual process.…”

Section: Other Measures Of Classification Accuracymentioning

confidence: 99%

Semi-automated text classification

Berardi

2014

SIGIR Forum

View full text Add to dashboard Cite

SommarioAl giorno d'oggi esiste una forte domanda di sistemi informatici per l'analisi automatica dei dati testuali. Le grandi industrie e organizzazioni hanno bisogno di elaborare enormi quantità di dati testuali, un'attività che non può essere eseguita con il solo lavoro umano. La Text Classification (TC)è l'attività di etichettare automaticamente i documenti testuali di un insieme D con le categorie tematiche di un insieme predefinito C. I sistemi di text classification lavorano con un'alta efficienza, ma non possono garantire un'accuratezza impeccabile dell'etichettatura.La Semi-Automated Text Classification (SATC) consiste nell'attività di ordinare un insieme di documenti testuali etichettati automaticamente D in modo che, se un annotatore umano validasse (i.e., ispezionasse e correggesse dove appropriato) una porzione dei primi documenti dell'ordinamento con l'obiettivo di incrementare l'accuratezza dell'etichettatura di D, l'incremento atteso venga massimizzato. Una strategia ovviaè quella di ordinare D in modo che i documenti che il classificatore ha etichettato con confidenza più bassa siano i primi dell'ordinamento. In questa tesi dimostriamo che questa strategiaè subottimale. Sviluppiamo nuovi metodi di ordinamento basati sulla teoria dell'utilità e sul concetto di guadagno della validazione, definito come il miglioramento dell'efficacia di classificazione che deriverebbe validando un dato documento etichettato automaticamente. Proponiamo inoltre nuove misure di efficacia per i metodi di ordinamento orientati alla SATC, basati sulla riduzione attesa dell'errore di classificazione, riduzione ottenuta dalla validazione di parte della lista do documenti generata da un dato metodo di ordinamento.Riportiamo i risultati degli esperimenti che dimostrano che, in confronto al metodo di base di cui sopra, e secondo le misure proposte, i nostri metodi di ordinamento basati sulla teoria dell'utilità sono in grado di ottenere una riduzione attesa dell'errore di classificazione sostanzialmente maggiore. Esploriamo quindi l'attività della SATC e il potenziale dei nostri metodi, in molteplici contesti della text classification. Questa tesiè, al meglio delle nostre conoscenze, la prima ad affrontare l'attività della semiautomated text classification. I II AbstractNowadays there is a high demand of information systems for the automatic analysis of textual data. Large industries and organizations need to process huge amounts of textual data, an activity that cannot be performed with human work only. Text Classification (TC) is the task of automatically labelling textual documents from a set D with thematic categories from a predefined set C. Text classification systems work with high efficiency, but they cannot guarantee impeccable labelling accuracy.Semi-Automated Text Classification (SATC) is the task of ranking a set D of automatically labelled textual documents in such a way that, if a human annotator validates (i.e., inspects and corrects where appropriate) the documents in a top-ranked portion of D with the goal o...

show abstract

Evaluation of information retrieval for E-discovery

Cited by 54 publications

References 32 publications

Using Recall and Elimination Terms in Separate Runs for High Volume Document Sorting

Using Recall and Elimination Terms in Separate Runs for High Volume Document Sorting

A utility-theoretic ranking method for semi-automated text classification

Semi-automated text classification

Contact Info

Product

Resources

About