Using neural networks to support high-quality evidence mapping

Røst, Thomas Brox; Slaughter, Laura; Nytrø, Øystein; Muller, Ashley Elizabeth; Vist, Gunn Elisabeth

doi:10.1186/s12859-021-04396-x

Cited by 5 publications

(1 citation statement)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The division is staffed by 50–60 employees and typically produces approximately 35–50 evidence synthesis products per year; this has roughly doubled under COVID-19. Based on early successful implementation with ML in COVID-19 reviews [ 4 , 17 – 19 ], a dedicated ML team was created in December of 2020 and tasked them with evaluating the time saving potential of existing ML functions, implementing those deemed successful, and horizon-scanning for new functions and applications. The team consisted of five systematic reviewers, one statistician, and one information specialist, with 3–10 years’ experience in evidence synthesis, and as of 2022, has grown to seven members and with a parallel informational specialist team [ 20 ].…”

Section: Introductionmentioning

confidence: 99%

Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system

Jardim

Rose

Ames

et al. 2022

BMC Med Res Methodol

View full text Add to dashboard Cite

Background Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers’ needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requires two trained reviewers. One function of RobotReviewer, an off-the-shelf machine learning system, is an automated risk of bias assessment. Methods We assessed the feasibility of adopting RobotReviewer within a national public health institute using a randomized, real-time, user-centered study. The study included 26 RCTs and six reviewers from two projects examining health and social interventions. We randomized these studies to one of two RobotReviewer platforms. We operationalized feasibility as accuracy, time use, and reviewer acceptability. We measured accuracy by the number of corrections made by human reviewers (either to automated assessments or another human reviewer’s assessments). We explored acceptability through group discussions and individual email responses after presenting the quantitative results. Results Reviewers were equally likely to accept judgment by RobotReviewer as each other’s judgement during the consensus process when measured dichotomously; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33). We were not able to compare time use. The acceptability of the program by researchers was mixed. Less experienced reviewers were generally more positive, and they saw more benefits and were able to use the tool more flexibly. Reviewers positioned human input and human-to-human interaction as superior to even a semi-automation of this process. Conclusion Despite being presented with evidence of RobotReviewer’s equal performance to humans, participating reviewers were not interested in modifying standard procedures to include automation. If further studies confirm equal accuracy and reduced time compared to manual practices, we suggest that the benefits of RobotReviewer may support its future implementation as one of two assessors, despite reviewer ambivalence. Future research should study barriers to adopting automated tools and how highly educated and experienced researchers can adapt to a job market that is increasingly challenged by new technologies.

show abstract

Section: Introductionmentioning

confidence: 99%

Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system

Jardim

Rose

Ames

et al. 2022

BMC Med Res Methodol

View full text Add to dashboard Cite

show abstract

Applications of artificial intelligence in battling against covid-19: A literature review

2021

Chaos, Solitons & Fractals

160

View full text Add to dashboard Cite

show abstract

A living critical interpretive synthesis to yield a framework on the production and dissemination of living evidence syntheses for decision-making

Mansilla,

Wang,

Piggott

et al. 2024

Implementation Sci

View full text Add to dashboard Cite

Background The COVID-19 pandemic has had an unprecedented impact in the global research production and has also increased research waste. Living evidence syntheses (LESs) seek to regularly update a body of evidence addressing a specific question. During the COVID-19 pandemic, the production and dissemination of LESs emerged as a cornerstone of the evidence infrastructure. This critical interpretive synthesis answers the questions: What constitutes an LES to support decision-making?; when should one be produced, updated, and discontinued?; and how should one be disseminated? Methods Searches included the Cochrane Library, EMBASE (Ovid), Health Systems Evidence, MEDLINE (Ovid), PubMed, and Web of Science up to 23 April 2024 and included articles that provide any insights on addressing the compass questions on LESs. Articles were selected and appraised, and their insights extracted. An interpretive and iterative coding process was used to identify relevant thematic categories and create a conceptual framework. Results Among the 16,630 non-duplicate records identified, 208 publications proved eligible. Most were non-empirical articles, followed by actual LESs. Approximately one in three articles were published in response to the COVID-19 pandemic. The conceptual framework addresses six thematic categories: (1) what is an LES; (2) what methodological approaches facilitate LESs production; (3) when to produce an LES; (4) when to update an LES; (5) how to make available the findings of an LES; and (6) when to discontinue LES updates. Conclusion LESs can play a critical role in reducing research waste and ensuring alignment with advisory and decision-making processes. This critical interpretive synthesis provides relevant insights on how to better organize the global evidence architecture to support their production. Trial registration PROSPERO registration: CRD42021241875.

show abstract

Using neural networks to support high-quality evidence mapping

Cited by 5 publications

References 23 publications

Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system

Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system

Applications of artificial intelligence in battling against covid-19: A literature review

A living critical interpretive synthesis to yield a framework on the production and dissemination of living evidence syntheses for decision-making

Contact Info

Product

Resources

About