Robust Statistical Stopping Criteria for Automated Screening in Systematic Reviews

Abstract

Active learning for systematic review screening promises to reduce the human effort required to identify relevant documents for a systematic review. Machines and humans work together, with humans providing training data, and the machine optimising the documents that the humans screen. This enables the identification of all relevant documents after viewing only a fraction of the total documents. However, current approaches lack robust stopping criteria, so that reviewers do not know when they have seen all or a certain proportion of relevant documents. This means that such systems are hard to implement in live reviews. This paper introduces a workflow with robust and flexible statistical stopping criteria, which offer real work reductions on the basis of a given confidence level of reaching a given recall. The stopping criteria are shown on test datasets to achieve a reliable level of recall, while still providing work reductions of on average 17\%. Other methods proposed previously are shown to provide inconsistent recall and work reductions across datasets.

Climate action for health and wellbeing in cities: a protocol for the systematic development of a database of peer-reviewed studies using machine learning methods

“…We will use supervised machine learning (ML) to accelerate the screening process through an iterative cycle of training, testing against manual classification and re-training of the ML algorithm until a stopping rule is met ( Callaghan & Müller-Hansen, 2019 ). We will use NACSOS research platform for screening process and Python scikit-learn library for machine learning ( Callaghan et al ., 2020 ; Pedregosa et al , 2012 ).…”

On the use of computer‐assistance to facilitate systematic mapping

“…"> Screening —Machine learning tools can help to increase efficiency in screening by prioritising relevant research so that eligible studies can be found early on in the screening process and other stages (meta‐data extraction) can begin sooner (O'Mara‐Eves et al, 2015), although traditionally, all records should be screened by at least one human (CEE, 2018). Recent methodological developments have suggested processes of transparently cutting off screening processes at given levels of confidence about the completeness of the process (Callaghan & Müller‐Hansen, 2019). In addition, named entity recognition can be used to identify terms from a predefined dictionary or taxonomy to support screening: for example, Lamb et al (2019) used a dictionary of city names to identify case studies from the climate literature. …”