Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review

Abstract

Objective To assess the methodological quality of studies on prediction models developed using machine learning techniques across all medical specialties. Design Systematic review. Data sources PubMed from 1 January 2018 to 31 December 2019. Eligibility criteria Articles reporting on the development, with or without external validation, of a multivariable prediction model (diagnostic or prognostic) developed using supervised machine learning for individualised predictions. No restrictions applied for study design, data source, or predicted patient related health outcomes. Review methods Methodological quality of the studies was determined and risk of bias evaluated using the prediction risk of bias assessment tool (PROBAST). This tool contains 21 signalling questions tailored to identify potential biases in four domains. Risk of bias was measured for each domain (participants, predictors, outcome, and analysis) and each study (overall). Results 152 studies were included: 58 (38%) included a diagnostic prediction model and 94 (62%) a prognostic prediction model. PROBAST was applied to 152 developed models and 19 external validations. Of these 171 analyses, 148 (87%, 95% confidence interval 81% to 91%) were rated at high risk of bias. The analysis domain was most frequently rated at high risk of bias. Of the 152 models, 85 (56%, 48% to 64%) were developed with an inadequate number of events per candidate predictor, 62 handled missing data inadequately (41%, 33% to 49%), and 59 assessed overfitting improperly (39%, 31% to 47%). Most models used appropriate data sources to develop (73%, 66% to 79%) and externally validate the machine learning based prediction models (74%, 51% to 88%). Information about blinding of outcome and blinding of predictors was, however, absent in 60 (40%, 32% to 47%) and 79 (52%, 44% to 60%) of the developed models, respectively. Conclusion Most studies on machine learning based prediction models show poor methodological quality and are at high risk of bias. Factors contributing to risk of bias include small study size, poor handling of missing data, and failure to deal with overfitting. Efforts to improve the design, conduct, reporting, and validation of such studies are necessary to boost the application of machine learning based prediction models in clinical practice. Systematic review registration PROSPERO CRD42019161764.

Machine Learning Risk Prediction Model of 90-day Mortality After Gastrectomy for Cancer

“…Machine learning approaches, a branch of artificial intelligence, have evolved and grown popularity because they have greater flexibility to capture complex nonlinear relationships especially when a very large number and loose data are used 12. New technologies, however, should follow the methodological standards of data treatment, avoiding common pitfalls such as insufficient sample size, risk of bias, unclear descriptions of treatment and patients’ characteristics, handling of missing data or lack of use of calibration models 13,14…”

Passive Sensing in the Prediction of Suicidal Thoughts and Behaviors: Protocol for a Systematic Review

“…To assess the risk of bias at the study level, 2 independent reviewers will use the signaling questions of the Prediction Model Risk of Bias Assessment Tool (PROBAST) [ 40 ]. According to the tool, 4 domains will be analyzed: participants, predictors, outcome, and analysis.…”

Accuracy of Automated Computer-Aided Diagnosis for Stroke Imaging: A Critical Evaluation of Current Evidence

“…Bias emerging during the development of any automated system is common in AI studies 57 and can reflect issues with the underlying datasets or model development techniques. Numerous examples of biases have emerged, including those related to sex, 58 race, 59 and geography, 60 which could inadvertently exacerbate underlying health care inequalities.…”