A real-world evaluation of the implementation of NLP technology in abstract screening of a systematic review

Abstract

The laborious and time-consuming nature of systematic reviews hinders the dissemination of up-to-date evidence synthesis. Well-performing natural language processing (NLP) for systematic reviews have been developed, showing promise to improve efficiency. However, the feasibility and value of these tools have not been comprehensively demonstrated in a real-world review. We developed an NLP-assisted abstract screening tool that provides text inclusion recommendations, keyword highlights, and visual context cues. We evaluated this tool in a living systematic review on SARS-CoV-2 seroprevalence, conducting a quality improvement assessment of screening with and without the tool. We evaluated changes to abstract screening speed, screening accuracy, characteristics of included texts, and user satisfaction. The tool improved efficiency, reducing screening time per abstract by 45.9% and decreasing inter-reviewer conflict rates. The tool conserved precision of article inclusion (positive predictive value; 0.92 with tool vs 0.88 without) and recall (sensitivity; 0.90 vs 0.81). The summary statistics of included studies were similar with and without the tool. Users were satisfied with the tool (mean satisfaction score of 4.2/5). We evaluated an abstract screening process where one human reviewer was replaced with the tool's votes, finding that this maintained recall (0.92 one-person, one-tool vs 0.90 two tool-assisted humans) and precision (0.91 vs 0.92) while reducing screening time by 70%. Implementing an NLP tool in this living systematic review improved efficiency, maintained accuracy, and was well-received by researchers, demonstrating the real-world effectiveness of NLP in expediting evidence synthesis.

Efficiency and Workload Reduction of Semi-automated Citation Screening Software for Creating Clinical Practice Guidelines: A Prospective Observational Study

“…An alternative inquiry utilizing another natural language processing tool, Covidence, reported sensitivity and PPVs of 0.90 and 0.92, respectively. 18 Here, we demonstrated that the sensitivity of the automated citation screening in the primary analysis varied from 0.2 to 0.75, which is lower than that found in the previous study that investigated the accuracy of ASReview. 4 The difference can be explained by the setting of the standard reference: whereas the previous report set the final list of included studies as the standard reference, we used the results of the conventional method after the first screening as the standard reference in the primary analysis.…”

“…[15][16][17] Other publications employing similar software have indicated the feasibility of replacing the conventional method with an efficient automated approach for screening studies in a systematic review. 18,19 Among several software packages that use natural language processing tools, the reliance on prior training data input varies. 6,9 For instance, DistillerAI requires a training dataset comprising 40 excluded and 10 included references for active machine learning.…”

Prompting is all you need: LLMs for systematic review screening

“…We calibrated our reviewers according to screening proficiency by having prospective reviewers first screen a ‘calibration set’ of abstracts. This set was sourced from a prior study by the SeroTracker group, 54 which assessed the performance of dual human reviewer workflows. Notably, the SeroTracker researchers were experienced SR screeners, having contributed to the SeroTracker living SR for over a year, and represent a high-performing baseline for screening accuracy.…”

“…For full-texts, we counted the total number of ‘Included’ and ‘Excluded’ articles from Covidence. Based on previous studies, the time required to screen a single abstract ranges from 20-461 seconds, 5,54,55 and 4.3-20 minutes for a single full-text article. 55,56 We aligned with Perlman-Arrow et al 54 due to our use of the same ST dataset and set the screening time at 30 seconds per abstract and 10 minutes per full-text.…”

“…For a representative comparison of GPT4-0125-preview and human screening performance, we used stratified random probability sampling across the four Oxford CEBM review questions, selecting one SR for each type. Our sample included various datasets: the SeroTracker dataset for reviews of prevalence (adapted from Perlman-Arrow et al 54 ), the Reinfection dataset for reviews of intervention benefits, the PA-Testing dataset for reviews of diagnostic test accuracy, and SVCF dataset for reviews of prognosis. Due to the high volume of SR studies focusing on intervention benefits, we also included the PA-Outcomes dataset in our sample through additional random sampling.…”