Applying machine classifiers to update searches: Analysis from two case studies

Abstract

Manual screening of citation records could be reduced by using machine classifiers to remove records of very low relevance. This seems particularly feasible for update searches, where a machine classifier can be trained from past screening decisions. However, feasibility is unclear for broad topics. We evaluate the performance and implementation of machine classifiers for update searches of public health research using two case studies. The first study evaluates the impact of using different sets of training data on classifier performance, comparing recall and screening reduction with a manual screening ‘gold standard’. The second study uses screening decisions from a review to train a classifier that is applied to rank the update search results. A stopping threshold was applied in the absence of a gold standard. Time spent screening titles and abstracts of different relevancy‐ranked records was measured. Results: Study one: Classifier performance varies according to the training data used; all custom‐built classifiers had a recall above 93% at the same threshold, achieving screening reductions between 41% and 74%. Study two: applying a classifier provided a solution for tackling a large volume of search results from the update search, and screening volume was reduced by 61%. A tentative estimate indicates over 25 h screening time was saved. In conclusion, custom‐built machine classifiers are feasible for reducing screening workload from update searches across a range of public health interventions, with some limitation on recall. Key considerations include selecting a training dataset, agreeing stopping thresholds and processes to ensure smooth workflows.

The effect of machine learning tools for evidence synthesis on resource use and time-to-completion: protocol for a retrospective pilot study

“…ML offers the potential to reduce resource use, produce evidence syntheses in less time, and maintain or perhaps exceed the current expectations of transparency, reproducibility, and methodological rigor. One example is the training of binary classifiers to predict the relevance of unread studies without human assessment: Aum and Choe recently used a classifier to predict systematic review study designs [18], Stansfield and colleagues to update living reviews [19], and Verdugo-Paiva and colleagues to update an entire COVID-19 database [20].…”

Managing the evidence infodemic: Automation approaches used for developing NICE COVID-19 living guidelines

“…The machine learning classifier building tool that is implemented as part of the EPPI-4 reviewer software [2] was used to train a custom COVID-19 classifier that distinguished between relevant and irrelevant studies in the surveillance search. The EPPI-4 machine learning classifier building tool (hereafter called the EPPI-4 custom classifier) implements a logistic regression model that utilises a "bag-of-words" approach using tri-grams without word stemming [3]. The EPPI-4 custom classifier was trained and tested in three cycles, using the cumulative search results and screening decisions from March 2020 to December 2021.…”

The effect of machine learning tools for evidence synthesis on resource use and time-to-completion: protocol for a retrospective pilot study

“…ML offers the potential to reduce resource use, produce evidence syntheses in less time, and maintain or perhaps exceed current expectations of transparency, reproducibility, and methodological rigor. One example is the training of binary classi ers to predict the relevance of unread studies without human assessment: Aum and Choe recently used a classi er to predict systematic review study designs [16], Stans eld and colleagues to update living reviews [17], and Verdugo-Paiva and colleagues to update an entire COVID-19 database [18]. ML tools have been available for systematic reviewers for at least ten years, yet uptake has been slow.…”