A Machine Learning Framework Reduces the Manual Workload for Systematic Reviews of the Diagnostic Performance of Prostate Magnetic Resonance Imaging

Nedelcu, Andrea; Oerther, Benedict; Engel, Hannes; Sigle, August; Schmucker, Christine; Schoots, Ivo G.; Eisenblätter, Michel; Benndorf, Matthias

doi:10.1016/j.euros.2023.07.005

Cited by 2 publications

(1 citation statement)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Research demonstrates that AL can significantly reduce the workload involved in screening records, as evidenced by studies that report marked time savings [ 18 , 24 ]. Moreover, AL facilitates the inclusion of a broader array of papers during the search phase, a benefit detailed in the work of [ 6 ], where a team screened over 11,000 most likely relevant papers out of a set of 165.046 hits to find out what factors and interaction of factors contribute to the onset, maintenance, and relapse of anxiety-, depressive-, and substance use disorders.…”

Section: Introductionmentioning

confidence: 99%

Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study

Byrne,

Hofstee,

Teijema

et al. 2024

Syst Rev

View full text Add to dashboard Cite

Software that employs screening prioritization through active learning (AL) has accelerated the screening process significantly by ranking an unordered set of records by their predicted relevance. However, failing to find a relevant paper might alter the findings of a systematic review, highlighting the importance of identifying elusive papers. The time to discovery (TD) measures how many records are needed to be screened to find a relevant paper, making it a helpful tool for detecting such papers. The main aim of this project was to investigate how the choice of the model and prior knowledge influence the TD values of the hard-to-find relevant papers and their rank orders. A simulation study was conducted, mimicking the screening process on a dataset containing titles, abstracts, and labels used for an already published systematic review. The results demonstrated that AL model choice, and mostly the choice of the feature extractor but not the choice of prior knowledge, significantly influenced the TD values and the rank order of the elusive relevant papers. Future research should examine the characteristics of elusive relevant papers to discover why they might take a long time to be found.

show abstract

Section: Introductionmentioning

confidence: 99%

Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study

Byrne,

Hofstee,

Teijema

et al. 2024

Syst Rev

View full text Add to dashboard Cite

show abstract

Machine learning to optimize literature screening in medical guideline development

Harmsen,

de Groot,

Harkema

et al. 2024

Syst Rev

View full text Add to dashboard Cite

Objectives In a time of exponential growth of new evidence supporting clinical decision-making, combined with a labor-intensive process of selecting this evidence, methods are needed to speed up current processes to keep medical guidelines up-to-date. This study evaluated the performance and feasibility of active learning to support the selection of relevant publications within medical guideline development and to study the role of noisy labels. Design We used a mixed-methods design. Two independent clinicians’ manual process of literature selection was evaluated for 14 searches. This was followed by a series of simulations investigating the performance of random reading versus using screening prioritization based on active learning. We identified hard-to-find papers and checked the labels in a reflective dialogue. Main outcome measures Inter-rater reliability was assessed using Cohen’s Kappa (ĸ). To evaluate the performance of active learning, we used the Work Saved over Sampling at 95% recall (WSS@95) and percentage Relevant Records Found at reading only 10% of the total number of records (RRF@10). We used the average time to discovery (ATD) to detect records with potentially noisy labels. Finally, the accuracy of labeling was discussed in a reflective dialogue with guideline developers. Results Mean ĸ for manual title-abstract selection by clinicians was 0.50 and varied between − 0.01 and 0.87 based on 5.021 abstracts. WSS@95 ranged from 50.15% (SD = 17.7) based on selection by clinicians to 69.24% (SD = 11.5) based on the selection by research methodologist up to 75.76% (SD = 12.2) based on the final full-text inclusion. A similar pattern was seen for RRF@10, ranging from 48.31% (SD = 23.3) to 62.8% (SD = 21.20) and 65.58% (SD = 23.25). The performance of active learning deteriorates with higher noise. Compared with the final full-text selection, the selection made by clinicians or research methodologists deteriorated WSS@95 by 25.61% and 6.25%, respectively. Conclusion While active machine learning tools can accelerate the process of literature screening within guideline development, they can only work as well as the input given by human raters. Noisy labels make noisy machine learning.

show abstract

A Machine Learning Framework Reduces the Manual Workload for Systematic Reviews of the Diagnostic Performance of Prostate Magnetic Resonance Imaging

Cited by 2 publications

References 9 publications

Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study

Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study

Machine learning to optimize literature screening in medical guideline development

Contact Info

Product

Resources

About