Comparing machine and human reviewers to evaluate the risk of bias in randomized controlled trials

Abstract

Background: Evidence from new health technologies is growing, along with demands for evidence to inform policy decisions, creating challenges in completing health technology assessments (HTAs)/systematic reviews (SRs) in a timely manner. Software can decrease the time and burden by automating the process, but evidence validating such software is limited. We tested the accuracy of RobotReviewer, a semi-autonomous risk of bias (RoB) assessment tool, and its agreement with human reviewers.Methods: Two reviewers independently conducted RoB assessments on a sample of randomized controlled trials (RCTs), and their consensus ratings were compared with those generated by RobotReviewer. Agreement with the human reviewers was assessed using percent agreement and weighted kappa (κ). The accuracy of RobotReviewer was also assessed by calculating the sensitivity, specificity, and area under the curve in comparison to the consensus agreement of the human reviewers. Results:The study included 372 RCTs. Inter-rater reliability ranged from κ = −0.06 (no agreement) for blinding of participants and personnel to κ = 0.62 (good agreement) for random sequence generation (excluding overall RoB).RobotReviewer was found to use a high percentage of "irrelevant supporting quotations" to complement RoB assessments for blinding of participants and personnel (72.6%), blinding of outcome assessment (70.4%), and allocation concealment (54.3%). Conclusion:RobotReviewer can help with risk of bias assessment of RCTs but cannot replace human evaluations. Thus, reviewers should check and validate RoB assessments from RobotReviewer by consulting the original article when not relevant supporting quotations are provided by RobotReviewer. This consultation is in line with the recommendation provided by the developers. K E Y W O R D Sartificial intelligence, health technology assessment (HTA), inter-rater reliability, randomized controlled trial, risk of bias, systematic review | INTRODUCTIONKnowledge synthesis products such as health technology assessments (HTAs) and systematic reviews (SRs) are important sources of high-quality information for policy makers, and the production of these assessments and reports requires expertise, human resources, and time. 1 Given the proliferation of health technologies and

How to keep up to date with medical information using web‐based resources: a systematised review and narrative synthesis

“…However, in areas with a degree of subjectivity (e.g. determining whether a study is at risk of bias), the subjective opinion of an expert human may be still more reliable than a machine’s decision (Armijo‐Olivo, Craig, & Campbell, 2020; Marshall & Wallace, 2019).…”

Rapid review methods series: Guidance on the use of supportive software

“…However, the Cochrane RoB 1.0 tool was updated and replaced by the RoB 2.0 tool 55. RobotReviewer’s accuracy has been tested by the developers50 and three independent research teams 7 56 57. The authors reported moderate agreement between the tool and human reviewers (range: 46%–100%) 7 50 56 57.…”

“…RobotReviewer’s accuracy has been tested by the developers50 and three independent research teams 7 56 57. The authors reported moderate agreement between the tool and human reviewers (range: 46%–100%) 7 50 56 57. RobotReviewer7 50 56 57 can be helpful in supporting a reviewer’s RoB assessment by providing an automated assessment of four out of seven domains, which can save time and costs.…”

Automation of systematic reviews of biomedical literature: a systematic review of studies indexed in PubMed

“…Following the Cochrane Risk of Bias tool for randomised controlled trials (RCTs), RobotReviewer provides an overall assessment of risk of bias, and extracts supporting sentences from PDF les of full-text reports 67 . RobotReviewer was used in further ve studies 18,99,103,119,137 . One paper assessed the risk of bias in preclinical animal studies, comparing various techniques including recurrent neural networks with attention, convolutional neural networks and BERT 150 .…”