An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes

Abstract

Background Systematic reviews often require substantial resources, partially due to the large number of records identified during searching. Although artificial intelligence may not be ready to fully replace human reviewers, it may accelerate and reduce the screening burden. Using DistillerSR (May 2020 release), we evaluated the performance of the prioritization simulation tool to determine the reduction in screening burden and time savings. Methods Using a true recall @ 95%, response sets from 10 completed systematic reviews were used to evaluate: (i) the reduction of screening burden; (ii) the accuracy of the prioritization algorithm; and (iii) the hours saved when a modified screening approach was implemented. To account for variation in the simulations, and to introduce randomness (through shuffling the references), 10 simulations were run for each review. Means, standard deviations, medians and interquartile ranges (IQR) are presented. Results Among the 10 systematic reviews, using true recall @ 95% there was a median reduction in screening burden of 47.1% (IQR: 37.5 to 58.0%). A median of 41.2% (IQR: 33.4 to 46.9%) of the excluded records needed to be screened to achieve true recall @ 95%. The median title/abstract screening hours saved using a modified screening approach at a true recall @ 95% was 29.8 h (IQR: 28.1 to 74.7 h). This was increased to a median of 36 h (IQR: 32.2 to 79.7 h) when considering the time saved not retrieving and screening full texts of the remaining 5% of records not yet identified as included at title/abstract. Among the 100 simulations (10 simulations per review), none of these 5% of records were a final included study in the systematic review. The reduction in screening burden to achieve true recall @ 95% compared to @ 100% resulted in a reduced screening burden median of 40.6% (IQR: 38.3 to 54.2%). Conclusions The prioritization tool in DistillerSR can reduce screening burden. A modified or stop screening approach once a true recall @ 95% is achieved appears to be a valid method for rapid reviews, and perhaps systematic reviews. This needs to be further evaluated in prospective reviews using the estimated recall.

Evaluating the relationship between citation set size, team size and screening methods used in systematic reviews: a cross-sectional study

“…In addition to evaluating whether investigators performing large reviews adapt through team size, we also evaluated the frequency of application of other methodologies intended to reduce workload. Over the past two decades, there has been considerable interest in the ability of natural language processing to assist with abstract screening, and available evidence suggests that with the proper application the human screening burden can be reduced and time saved [ 43 , 44 ]. Yet, despite a significant number of publications on the topic, and incorporation into a number of common screening platforms, only one of the 259 SRs reported its application.…”

Utilizing Artificial Intelligence to Manage COVID-19 Scientific Evidence Torrent with Risklick AI: A Critical Tool for Pharmacology and Therapy Development

“…All clinical trials and scientific publications were analyzed and verified manually. To optimize the result comparison between the different search tools, recall (the number of positive class predictions made out of all positive examples in the dataset), precision (the number of positive class predictions that actually belong to the positive class), and F1 score (single score that balances both the concerns of precision and recall in one number) were calculated [ 22 ].…”

Guidance for using artificial intelligence for title and abstract screening while conducting knowledge syntheses

“…Several studies have been published since 2015 using and evaluating the use of AI and prioritized screening, many with encouraging results [ 10 , 25 – 37 ]. For example, to identify 95% of the studies included at the title and abstract level, studies have reported a reduction in the number of records that need to be screened of 40% [ 32 ] and 47.1% [ 34 ].…”

“…A review by O’Mara-Eves in 2015 reported that several studies evaluated machine-learning for reducing the in workload for screening records, but noted that there is little overlap between the outcomes (e.g., recall of 95% vs retrieving all relevant studies), making it difficult to conclude which approach is best [ 1 ]. More recent studies have generally concluded that full automation (level 4 automation; see Glossary of Terms) performs poorly, while semi-automation (level 2 automation) may be more reliable [ 10 , 30 , 33 , 34 ]. Although AI is not currently suitable to fully replace humans in title and abstract screening, there is value to be gained from AI use and some basic principles for teams who produce knowledge synthesis products to adopt are needed.…”

“…With no current consensus on how to best use AI for study selection, and several studies published in the area performance of AI and AML [ 26 , 27 , 30 – 32 , 34 ], many researchers are may be interested by the premise, but are uncertain as to its validity and means for operationalization. As some of the barriers to adoption of new technologies are the challenges in set-up [ 3 ], we have provided different situations and considerations for knowledge synthesis teams to consider when using AI for title and abstract screening while conducting reviews.…”