Automated medical literature screening using artificial intelligence: a systematic review and meta-analysis

Abstract

Objective We aim to investigate the application and accuracy of artificial intelligence (AI) methods for automated medical literature screening for systematic reviews. Materials and Methods We systematically searched PubMed, Embase, and IEEE Xplore Digital Library to identify potentially relevant studies. We included studies in automated literature screening that reported study question, source of dataset, and developed algorithm models for literature screening. The literature screening results by human investigators were considered to be the reference standard. Quantitative synthesis of the accuracy was conducted using a bivariate model. Results Eighty-six studies were included in our systematic review and 17 studies were further included for meta-analysis. The combined recall, specificity, and precision were 0.928 [95% confidence interval (CI), 0.878–0.958], 0.647 (95% CI, 0.442–0.809), and 0.200 (95% CI, 0.135–0.287) when achieving maximized recall, but were 0.708 (95% CI, 0.570–0.816), 0.921 (95% CI, 0.824–0.967), and 0.461 (95% CI, 0.375–0.549) when achieving maximized precision in the AI models. No significant difference was found in recall among subgroup analyses including the algorithms, the number of screened literatures, and the fraction of included literatures. Discussion and Conclusion This systematic review and meta-analysis study showed that the recall is more important than the specificity or precision in literature screening, and a recall over 0.95 should be prioritized. We recommend to report the effectiveness indices of automatic algorithms separately. At the current stage manual literature screening is still indispensable for medical systematic reviews.

Are Machines-learning Methods More Efficient than Humans in Triaging Literature for Systematic Reviews?

“…In addition, both methods performed well in the binary classification of records, but not as much for providing the reasons for exclusion using the pre-specified PICOS criteria. In a recent meta-analyses of AI algorithms for automated searches, the combined recall, specificity, and precision values were 0.928 [95% confidence interval (CI), 0.878– 0.958], 0.647 (95% CI, 0.442–0.809), and 0.200(95% CI, 0.135–0.287) when achieving maximized recall, and were 0.708 (95% CI, 0.570–0.816), 0.921 (95% CI, 0.824–0.967), and 0.461 (95% CI, 0.375–0.549) when achieving maximized precision [19]. The approach used in the present study was not designed to optimize either recall or precision; however, the values of recall (0.78 to 0.86), specificity (0.85 to 0.90), and precision (0.33 to 0.74) obtained with the binary classification of records in the present study are within the range of values reported by Feng et al [19].…”

“…SVM was considered separately because evidence suggested that SVM classifiers had the best performance for text classification [19]. There were no significant differences in recall, specificity, and precision between SVM and other algorithms collectively [19]. In the present study, fine-tuned BERTs and SVMs were applied to the title and abstract screening phase of two separate clinical SLRs and the results compared with those obtained by humans.…”

“…The use of a multikernel SVM and automatic parameterization markedly improved the speed of training an SVM to manage highly dimensional data with good accuracy [18]. A recent SLR and meta-analysis of publications addressing AI methods used for automated medical literature screening included 86 studies in the review, and 71 in the meta-analysis [19]. This study found that the most commonly used classifier was SVM.…”

“…; naive Bayes, K-nearest neighbor, perceptron, random forest, convolutional neural networks, radial basis function kernel) and SVM. SVM was considered separately because evidence suggested that SVM classifiers had the best performance for text classification [19]. There were no significant differences in recall, specificity, and precision between SVM and other algorithms collectively [19].…”

Machine learning for accelerating screening in evidence reviews

“…Despite the large number of available tools and evaluations of their accuracy, little has been published to bring together these evaluations and nothing to compare the accuracy of tools. An extensive systematic review [31] synthesized tool accuracy across studies but made no attempt to compare the accuracy of different tools. This type of comparative assessment is currently limited by the heterogeneity of methods used in different studies.…”