An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews

Kusa, Wojciech; Lipani, Aldo; Knoth, Petr; Hanbury, Allan

doi:10.1016/j.iswa.2023.200193

Cited by 9 publications

(7 citation statements)

References 71 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Automated citation screening is an umbrella term for using NLP, machine learning and information retrieval (IR) techniques with the goal of decreasing the time spent on manual screening. Classification approaches train a supervised model on an annotated dataset to determine whether a paper should be included or excluded from the review [23,24].…”

Section: Automated Citation Screeningmentioning

confidence: 99%

CRUISE-Screening: Living Literature Reviews Toolbox

Kusa,

Knoth,

Hanbury

2023

Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

Self Cite

View full text Add to dashboard Cite

Keeping up with research and finding related work is still a timeconsuming task for academics. Researchers sift through thousands of studies to identify a few relevant ones. Automation techniques can help by increasing the efficiency and effectiveness of this task. To this end, we developed CRUISE-Screening, a web-based application for conducting living literature reviews -a type of literature review that is continuously updated to reflect the latest research in a particular field. CRUISE-Screening is connected to several search engines via an API, which allows for updating the search results periodically. Moreover, it can facilitate the process of screening for relevant publications by using text classification and question answering models. CRUISE-Screening can be used both by researchers conducting literature reviews and by those working on automating the citation screening process to validate their algorithms. The application is open-source, 1 and a demo is available under this URL: https://citation-screening.ec.tuwien.ac.at. We discuss the limitations of our tool in Appendix A.

show abstract

Section: Automated Citation Screeningmentioning

confidence: 99%

CRUISE-Screening: Living Literature Reviews Toolbox

Kusa,

Knoth,

Hanbury

2023

Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

Self Cite

View full text Add to dashboard Cite

show abstract

“…Several evaluation measures have been proposed to measure the effectiveness of TAR systems [21]. One popular approach is to evaluate the system's precision at a fixed recall level (Precision at r% recall, 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑟 %), representing the percentage of relevant retrieved documents [13,15]. Another approach is to evaluate the work saved compared to the random ordering of the documents, using Work Saved over Sampling at r% recall evaluation measure (𝑊 𝑆𝑆@𝑟 %) [3].…”

Section: Introductionmentioning

confidence: 99%

VoMBaT: A Tool for Visualising Evaluation Measure Behaviour in High-Recall Search Tasks

Kusa

Lipani

Knoth

et al. 2023

Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

View full text Add to dashboard Cite

The objective of High-Recall Information Retrieval (HRIR) is to retrieve as many relevant documents as possible for a given search topic. One approach to HRIR is Technology-Assisted Review (TAR), which uses information retrieval and machine learning techniques to aid the review of large document collections. TAR systems are commonly used in legal eDiscovery and systematic literature reviews. Successful TAR systems are able to find the majority of relevant documents using the least number of assessments. Commonly used retrospective evaluation assumes that the system achieves a specific, fixed recall level first, and then measures the precision or work saved (e.g., precision at r% recall). This approach can cause problems related to understanding the behaviour of evaluation measures in a fixed recall setting. It is also problematic when estimating time and money savings during technology-assisted reviews.This paper presents a new visual analytics tool to explore the dynamics of evaluation measures depending on recall level. We implemented 18 evaluation measures based on the confusion matrix terms, both from general IR tasks and specific to TAR. The tool allows for a comparison of the behaviour of these measures in a fixed recall evaluation setting. It can also simulate savings in time and money and a count of manual vs automatic assessments for different datasets depending on the model quality. The tool is open-source, and the demo is available under the following URL: https://vombat.streamlit.app. CCS CONCEPTS• Information systems → Evaluation of retrieval results; Retrieval effectiveness; • Human-centered computing → Visual analytics.

show abstract

“…We show the WSS at 95% sensitivity (WSS@95%, solid red line). The formal definition of WSS@95% is: ( n true negative studies + n false negative studies)/ n all studies − 0.05, at 95% sensitivity [9] . …”

mentioning

confidence: 99%

“…This sums to 7.4–105.6 h for two reviewers. WSS@95% generally provides a good balance between screening sensitivity and work saved [9] . A systematic review of meta-analyses finds only a negligible effect on the overall results when a small percentage of relevant studies are missing [10] , which in our view contributes to consideration of WSS@95% as a reasonable outcome.…”

mentioning

confidence: 99%

See 1 more Smart Citation

A Machine Learning Framework Reduces the Manual Workload for Systematic Reviews of the Diagnostic Performance of Prostate Magnetic Resonance Imaging

Nedelcu,

Oerther,

Engel

et al. 2023

European Urology Open Science

View full text Add to dashboard Cite

An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews

Cited by 9 publications

References 71 publications

CRUISE-Screening: Living Literature Reviews Toolbox

CRUISE-Screening: Living Literature Reviews Toolbox

VoMBaT: A Tool for Visualising Evaluation Measure Behaviour in High-Recall Search Tasks

A Machine Learning Framework Reduces the Manual Workload for Systematic Reviews of the Diagnostic Performance of Prostate Magnetic Resonance Imaging

Contact Info

Product

Resources

About