Prioritising references for systematic reviews with RobotAnalyst: A user study

Abstract

Screening references is a time‐consuming step necessary for systematic reviews and guideline development. Previous studies have shown that human effort can be reduced by using machine learning software to prioritise large reference collections such that most of the relevant references are identified before screening is completed. We describe and evaluate RobotAnalyst, a Web‐based software system that combines text‐mining and machine learning algorithms for organising references by their content and actively prioritising them based on a relevancy classification model trained and updated throughout the process. We report an evaluation over 22 reference collections (most are related to public health topics) screened using RobotAnalyst with a total of 43 610 abstract‐level decisions. The number of references that needed to be screened to identify 95% of the abstract‐level inclusions for the evidence review was reduced on 19 of the 22 collections. Significant gains over random sampling were achieved for all reviews conducted with active prioritisation, as compared with only two of five when prioritisation was not used. RobotAnalyst's descriptive clustering and topic modelling functionalities were also evaluated by public health analysts. Descriptive clustering provided more coherent organisation than topic modelling, and the content of the clusters was apparent to the users across a varying number of clusters. This is the first large‐scale study using technology‐assisted screening to perform new reviews, and the positive results provide empirical evidence that RobotAnalyst can accelerate the identification of relevant studies. The results also highlight the issue of user complacency and the need for a stopping criterion to realise the work savings.

Robust Statistical Stopping Criteria for Automated Screening in Systematic Reviews

“…Some studies give the example of heuristic stopping criteria based on drawing a given number of irrelevant articles in a row [6,7]. We take this as a proxy for estimating that the proportion of documents remaining in the unseen documents is low.…”

“…This is a time consuming and repetitive task, and in a research environment with constrained resources and increasing amounts of literature, this may limit the scope of the evidence synthesis projects undertaken. Several papers have developed Active Learning (AL) approaches [3][4][5][6][7] to reduce the time required to screen documents. This paper sets out how current approaches are unsuitable in practice, and outlines and evaluates a small modification that would make AL systems ready for live reviews.…”

“…• Sampling criteria: Reviewers estimate the number of relevant documents by taking a random sample at the start of the process. They stop when this number, or a given proportion of it, has been reached [12] • Heuristics: Reviewers stop when a given number of irrelevant articles are seen in a row [6,7]. • Pragmatic criteria: Reviewers stop when they run out of time [3].…”

Enhancing WIKISTIM.org Using Machine Learning Approaches

“…RobotAnalyst is a Web-based screening system that leverages algorithms for information retrieval, text mining, natural language processing (NLP), and machine learning (ML) to assist reviewers in prioritizing references and exploring biomedical data bases using automatic terminology extraction, topic modeling, and descriptive clustering. In a recent study of the software, RobotAnalyst was associated with an up to 71% decrease in screening effort (4). WIKISTIM could then exploit semantic technologies, including NLP and ML techniques, to extract the appropriate datapoints from relevant identified papers, with a certain degree of human oversight required for this stage.…”

Statistical Stopping Criteria for Automated Screening in Systematic Reviews

“…This is a time consuming and repetitive task, and in a research environment with constrained resources and increasing amounts of literature, this may limit the scope of the evidence synthesis projects undertaken. Several papers have developed Active Learning (AL) approaches [3][4][5][6][7] to reduce the time required to screen documents. This paper sets out how current approaches are unreliable in practice, and outlines and evaluates modifications that would make AL systems ready for live reviews.…”