The SAFE Procedure: A Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses

Abstract

Active learning has become an increasingly popular method for screening large amounts of data in systematic reviews and meta-analyses. The active learning process continually improves its predictions on the remaining unlabeled records, with the goal of identifying all relevant records as early as possible. However, determining the optimal point at which to stop the active learning process is a challenge. The cost of additional labeling of records by the reviewer must be balanced against the cost of errors made by the current model. This paper introduces the SAFE procedure, a practical and conservative set of stopping heuristics that offers a clear guideline for determining when to end the active learning process in screening software like ASReview. Our main conclusion of this paper is that relying on a single stopping rule is not sufficient and employing an eclectic mix of stopping heuristics helps to minimize the risk of missing relevant papers in the screening processThe SAFE procedure combines different heuristics to avoid stopping too early and potentially missing relevant records. The SAFE procedure takes into account the model's accuracy and uncertainty, as well as the cost of continuing to label records. This procedure for using active learning in systematic literature review screening provides a practical and efficient approach that can save significant time and resources while ensuring a conservative approach to determining when to end the active learning process. The proposed stopping heuristic balances the costs of continued screening with the risk of missing relevant records, providing a practical solution for reviewers to make informed decisions on when to stop screening. The proposed method can assist researchers in identifying relevant records early, which can ultimately lead to improved evidence synthesis and decision-making in many fields.

Addressing the Challenges of Reconstructing Systematic Reviews Datasets V2

“…For further discussion and a heuristic for determining a more nuanced stopping rule, we refer readers to the works of (Bloodgood & Vijay-Shanker, 2014;Callaghan & Mueller-Hansen, 2020;Vlachos, 2008;Yang et al, 2021) who provide a comprehensive evaluation of this issue. Additionally, see (Boetje & van de Schoot, 2023) for a practical and high-quality heuristic.…”

The influence of active learning model and prior knowledge choice on how long it takes to find hard-to-find relevant papers: Examining the variability of the time to discovery and the stability of its rank-orders

“…For an unlabelled dataset, for example, after including the 37 th paper, it is unknown whether a 38 th relevant paper exists or if the count stops at 37. Therefore, one cannot halt the process right after including the 37 th paper, necessitating the screener to proceed based on a predetermined stopping rule (Boetje & Schoot, 2023;Callaghan & Mueller-Hansen, 2020;Gargon et al, 2019;Vlachos, 2008;Yang et al, 2021). Suppose one screens with a (naïve) stopping rule of 100 irrelevant papers in a row, but the last relevant paper is ranked 138 th .…”