Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records

Abstract

Background Conducting a systematic review demands a significant amount of effort in screening titles and abstracts. To accelerate this process, various tools that utilize active learning have been proposed. These tools allow the reviewer to interact with machine learning software to identify relevant publications as early as possible. The goal of this study is to gain a comprehensive understanding of active learning models for reducing the workload in systematic reviews through a simulation study. Methods The simulation study mimics the process of a human reviewer screening records while interacting with an active learning model. Different active learning models were compared based on four classification techniques (naive Bayes, logistic regression, support vector machines, and random forest) and two feature extraction strategies (TF-IDF and doc2vec). The performance of the models was compared for six systematic review datasets from different research areas. The evaluation of the models was based on the Work Saved over Sampling (WSS) and recall. Additionally, this study introduces two new statistics, Time to Discovery (TD) and Average Time to Discovery (ATD). Results The models reduce the number of publications needed to screen by 91.7 to 63.9% while still finding 95% of all relevant records (WSS@95). Recall of the models was defined as the proportion of relevant records found after screening 10% of of all records and ranges from 53.6 to 99.8%. The ATD values range from 1.4% till 11.7%, which indicate the average proportion of labeling decisions the researcher needs to make to detect a relevant record. The ATD values display a similar ranking across the simulations as the recall and WSS values. Conclusions Active learning models for screening prioritization demonstrate significant potential for reducing the workload in systematic reviews. The Naive Bayes + TF-IDF model yielded the best results overall. The Average Time to Discovery (ATD) measures performance of active learning models throughout the entire screening process without the need for an arbitrary cut-off point. This makes the ATD a promising metric for comparing the performance of different models across different datasets.

The influence of active learning model and prior knowledge choice on how long it takes to find hard-to-find relevant papers: Examining the variability of the time to discovery and the stability of its rank-orders

“…The present study has important implications for the field of AL-aided screening tools as it highlights the use of the TD metric to help locate and assess the variability of the hard-to-find relevant papers across different simulation set-ups. Previous research on the TD has examined the ATD (or average-simulation TD in the context of simulation studies) across different models and datasets (Ferdinands et al, 2023). In contrast, this study was the first to use the average-record-TD to locate the hard-to-find relevant papers across different models and prior knowledge.…”

Utilizing ChatGPT to select literature for meta-analysis shows workload reduction while maintaining a similar recall level as manual curation

“…Previous AI models utilized in systemic reviews used active learning to select the training dataset and returned all records ordered by a "similarity". 5 However, LLMs were trained to predict text that follows the input text. By doing so, LLMs could directly answer questions and return whether an input record meet provided criteria or not.…”

“…In our research, we didn't use metrics like Work Saved over Sampling (WSS) and Average Time to Discover (ATD), 5 which were commonly used to evaluate previous AI. This is because LARS works in a completely different way.…”

“…In general, the workflow of meta-analysis has the following steps: (1) define research question; (2) select literature databases and design search strategy; (3) screen records based on their titles and abstracts; (4) screen records based on full text of records; (5) extract and synthesis data. In the present study, we focused on incorporating ChatGPT into the third step of this workflow.…”

“…3 Human annotators are needed to train the AI model on dataset. 4,5 AI models could provide predicted probability for all records based on "similarity" between them. However, researchers still need to screen all of them.…”