Impact of Surrogate Assessments on High-Recall Retrieval

Roegiest, Adam; Cormack, Gordon V.; Clarke, Charles L. A.; Grossman, Maura R.

doi:10.1145/2766462.2767754

Cited by 13 publications

(12 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Its development was informed by experience from TREC Total Recall tracks [24,49] and also adopted by the CLEF Technology Assisted Reviews in Empirical Medicine Tracks [30][31][32]. The 𝑙𝑜𝑠𝑠 𝑒𝑟 measure is the sum of two components:…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…𝑟𝑒𝑐𝑎𝑙𝑙 𝑡 ), but we choose not to adjust the method to simplify comparison with previous work and because the relative error measure already captures information about the difference between the achieved and target recall. The second component, 𝑙𝑜𝑠𝑠 𝑒 , motivated by experience from the TREC 2015 Total Recall Track [49], is defined as:…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…The standard approach to identifying relevant literature is to construct a Boolean query designed to optimise recall over precision and manually screen the results, an expensive and time-consuming process that can involve manual assessment of tens of thousands of documents [43,53]. In the legal domain, electronic discovery (eDiscovery) is the identification of documents for legal purposes, such as disclosure in response to litigation [24,46,49] or to meet the requirements of freedom of information (FoI) legislation [8,42]. In eDiscovery, it is important to identify as many relevant documents as possible given the resources available to ensure compliance with legal obligations and avoid potential penalties.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Stopping Methods for Technology-assisted Reviews Based on Point Processes

Stevenson,

Bin-Hezam

2023

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

Technology Assisted Review (TAR), which aims to reduce the effort required to screen collections of documents for relevance, is used to develop systematic reviews of medical evidence and identify documents that must be disclosed in response to legal proceedings. Stopping methods are algorithms which determine when to stop screening documents during the TAR process, helping to ensure that workload is minimised while still achieving a high level of recall. This paper proposes a novel stopping method based on point processes, which are statistical models that can be used to represent the occurrence of random events. The approach uses rate functions to model the occurrence of relevant documents in the ranking and compares four candidates, including one that has not previously been used for this purpose (hyperbolic). Evaluation is carried out using standard datasets (CLEF e-Health, TREC Total Recall, TREC Legal), and this work is the first to explore stopping method robustness by reporting performance on a range of rankings of varying effectiveness. Results show that the proposed method achieves the desired level of recall without requiring an excessive number of documents to be examined in the majority of cases and also compares well against multiple alternative approaches.

show abstract

Section: Evaluation Metricsmentioning

confidence: 99%

Section: Evaluation Metricsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Stopping Methods for Technology-assisted Reviews Based on Point Processes

Stevenson,

Bin-Hezam

2023

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

show abstract

“…Traditionally, relevance feedback has been construed as a method to automate the query-formulation task envisioned by TREC: The user's assessment of the results from an initial query are provided to the search tool, which reformulates the query and presents a new set of results to the user, and so on. More recently, supervised machine-learning methods have been used to harness relevance feedback, with reported effectiveness apparently exceeding Voorhees' "practical upper bound," see e.g., [6,14,25].…”

Section: Theorymentioning

confidence: 99%

“…Previously published results report recall-precision breakeven scores on the order of 80% for BMI and related methods [7,8,11,25]. With one notable exception (discussed below), these results were derived using simulated feedback from an assumed-infallible user, and evaluated with respect to the same assumed-infallible gold standard.…”

Section: Prediction and Rationalementioning

confidence: 99%

Navigating Imprecision in Relevance Assessments on the Road to Total Recall

Cormack

Grossman

2017

Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

Technology-assisted review ("TAR") systems seek to achieve "total recall"; that is, to approach, as nearly as possible, the ideal of 100% recall and 100% precision, while minimizing human review effort. The literature reports that TAR methods using relevance feedback can achieve considerably greater than the 65% recall and 65% precision reported by Voorhees as the "practical upper bound on retrieval performance . . . since that is the level at which humans agree with one another" (Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, 2000). This work argues that in order to build-as well as to, evaluate-TAR systems that approach 100% recall and 100% precision, it is necessary to model human assessment, not as absolute ground truth, but as an indirect indicator of the amorphous property known as "relevance." The choice of model impacts both the evaluation of system effectiveness, as well as the simulation of relevance feedback. Models are presented that better fit available data than the infallible ground-truth model. These models suggest ways to improve TAR-system effectiveness so that hybrid human-computer systems can improve on both the accuracy and efficiency of human review alone. This hypothesis is tested by simulating TAR using two datasets: the TREC 4 AdHoc collection, and a dataset consisting of 401,960 email messages that were manually reviewed and classified by a single individual, Roger, in his official capacity as Senior State Records Archivist. The results using the TREC 4 data show that TAR achieves higher recall and higher precision than the assessments by either of two independent NIST assessors, and blind adjudication of the email dataset, conducted by Roger, more than two years after his original review, shows that he could have achieved the same recall and better precision, while reviewing substantially fewer than 401,960 emails, had he employed TAR in place of exhaustive manual review.

show abstract