Using artificial intelligence methods for systematic review in health sciences: A systematic review

Abstract

The exponential increase in published articles makes a thorough and expedient review of literature increasingly challenging. This review delineated automated tools and platforms that employ artificial intelligence (AI) approaches and evaluated the reported benefits and challenges in using such methods. A search was conducted in 4 databases (Medline, Embase, CDSR, and Epistemonikos) up to April 2021 for systematic reviews and other related reviews implementing AI methods. To be included, the review must use any form of AI method, including machine learning, deep learning, neural network, or any other applications used to enable the full or semi-autonomous performance of one or more stages in the development of evidence synthesis. Twelve reviews were included, using nine different tools to implement 15 different AI methods. Eleven methods were used in the screening stages of the review (73%). The rest were divided: two in data extraction (13%) and two in risk of bias assessment (13%). The ambiguous benefits of the data extractions, combined with the reported advantages from 10 reviews, indicating that AI platforms have taken hold with varying success in evidence synthesis. However, the results are qualified by the reliance on the self-reporting of the review authors. Extensive human validation still appears required at this stage in implementing AI methods, though further evaluation is required to define the overall contribution of such platforms in enhancing efficiency and quality in evidence synthesis.

(Semi)automated approaches to data extraction for systematic reviews and meta-analyses in social sciences: A living review protocol

“…Table 1 provides a summary of scoped studies. Where some efforts focused on software applications, or "tools" that perform or assist with systematic review tasks (Harrison et al, 2020;Scott et al, 2021), others directed attention to underlying methods or techniques (e.g., machine learning algorithms) or reviewed multiple categorizations (see Blaizot et al, 2022;Schmidt et al, 2021;O'Connor et al, 2019). O'Connor et al, 2019).…”

“…O'Mara-Eves et al (2015), for example, reported that text-mining techniques for classifying and prioritizing (i.e., ranking) relevant studies had undergone substantial methodological advancement, yet also highlighted that where assessment methods could be implemented with relatively high confidence in clinical research, much work was needed to determine how systems might perform in other disciplines. Other researchers similarly noted issues such as heterogeneity in testing and performance metrics(Blaizot et al, 2022;Jonnalagadda et al, 2015;Tsafnat et al, 2014) as well as risk of systemic biases resulting from inconsistent annotations in training corpora(Schmidt et al, 2021). Across projects reviewed, calls resounded for additional assessment of automation methods, including testing methods across different datasets and domains and testing the same datasets across different automation methods(Schmidt et al, 2021, O'Mara-Eves et al, 2015O'Connor et al, 2019;Jonnalagadda et al, 2015).…”

“…Note. AI = Artificial Intelligence; T&Ab = Title and Abstract; RCT = Randomized Control Trials, ICASR = International Collaboration for Automation of Systematic Reviews; SR = Systematic Review.Existing platforms are available to support research teams in a range of time-consuming manual tasks(Blaizot et al, 2022). Even with these expediencies, not all key activities within the overall review process have received equal attention in application and technique development(O'Connor et al, 2019;Scott et al, 2021).…”

The role of open research in improving the standards of evidence synthesis: current challenges and potential solutions in systematic reviews

“…55 Therefore, several methods and tools have been developed through the use of artificial intelligence (AI) in order to aid the automation of SRs. [55][56][57] AI includes machine learning (ML) which utilizes computer algorithms similar to logistic regression and natural language processing (NLP) which analyses a vast number of texts and extracts information. 56,58 Both ML and NPL are commonly implemented technologies used in the semi-automated conduction of SRs.…”

“…[55][56][57] AI includes machine learning (ML) which utilizes computer algorithms similar to logistic regression and natural language processing (NLP) which analyses a vast number of texts and extracts information. 56,58 Both ML and NPL are commonly implemented technologies used in the semi-automated conduction of SRs. 58 ML uses statistical predictive methods to calculate the likelihood that an article is relevant and is commonly used during the screening process whereas NPL analyses the semantic meaning and is used during the data extraction step of an SR. 59 Several automation tools have been developed that aim to execute the time-consuming and labour-intensive tasks of an SR such as search (e.g.…”