Recommendations for Reporting Machine Learning Analyses in Clinical Research

Stevens, Laura; Mortazavi, Bobak J.; Deo, Rahul C.; Curtis, Lesley H.; Kao, David

doi:10.1161/circoutcomes.120.006556

Cited by 175 publications

(159 citation statements)

References 32 publications

Supporting

Mentioning

157

Contrasting

Unclassified

Order By: Relevance

“…However extensions of these guidelines are required in order to be fully applicable to deep learning systems (Wynants et al (2020)). Many efforts are already being done in this direction: extension of TRIPOD and CONSORT/SPIRIT (TRIPOD-ML, Collins and Moons (2019)) and CONSORT-AI/SPIRIT-AI (Liu et al (2019)) statements are being developed, focused on model validation and clinical trials, respectively; considerations for critically appraising ML studies are given in Faes et al (2020); and reporting recommendations are given in Stevens et al (2020).…”

Section: Resultsmentioning

confidence: 99%

“…General considerations about clinical prediction model (Steyerberg (2009)) are as relevant in AI models as in linear regression models, although in the former case are much more difficult to address. Protocols for AI model development are being developed, in the meantime the minimum requirements for dataset description should be assessed (Collins and Moons (2019); Liu et al (2019); Faes et al (2020); Stevens et al (2020)).…”

Section: Discussionmentioning

confidence: 99%

“…However, there is growing awareness in the community that the presence of different sources of bias significantly decreases the overall generalisation ability of the models, leading to overestimated model performance reported in internal validation compared to evaluation on independent test data (Soneson et al (2014); Cohen et al (2020b); Zech et al (2018); Maguolo and Nanni (2020)). In addition, numerous journal editorials are calling for better development, evaluation and reporting practices of machine learning models aimed for clinical application (Mateen et al (2020); Nagendran et al (2020); Campbell et al (2020); The Lancet Digital Health (2020); O’Reilly-Shah et al (2020); The Lancet Digital Health (2019); Stevens et al (2020)). Underneath these caveats are the growing concerns about ethics and risk of harmful outcomes of using AI in medical applications (Campolo et al (2018); Geis et al (2019); Brady and Neri (2020)).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Public Covid-19 X-ray datasets and their impact on model bias - a systematic review of a significant problem

Cruz

Bossa²,

Soelter³

et al. 2021

Preprint

View full text Add to dashboard Cite

Computer-aided-diagnosis for COVID-19 based on chest X-ray suffers from weak bias assessment and limited quality-control. Undetected bias induced by inappropriate use of datasets and improper consideration of confounders prevents the translation of prediction models into clinical practice. This study provides a systematic evaluation of publicly available COVID-19 chest X-ray datasets, determining their potential use and evaluating potential sources of bias. Only 5 out of 256 identified datasets met at least the criteria for proper assessment of risk of bias and could be analysed in detail. Remarkably almost all of the datasets utilised in 78 papers published in peer-reviewed journals, are not among these 5 datasets, thus leading to models with a high risk of bias. This raises concerns about the suitability of such models for clinical use. This systematic review highlights the limited description of datasets employed for modelling and aids researchers to select the most suitable datasets for their task.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Public Covid-19 X-ray datasets and their impact on model bias - a systematic review of a significant problem

Cruz

Bossa²,

Soelter³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We were transparent and cautious in handling these issues, and we followed the recommendations recently published by the American Heath Association. 28 Our machine learning used the same set of patients initially used by Chiò and colleagues to define their subtypes. 23 It is unlikely that this led to recruitment bias as information concerning the Chiò subtypes was not used to generate our models, and the clinical parameters used to create the models are standard across the ALS field.…”

Section: Discussionmentioning

confidence: 99%

Identification and prediction of ALS subgroups using machine learning

Faghri

Brunn

Dadu

et al. 2021

Preprint

View full text Add to dashboard Cite

Background The disease entity known as amyotrophic lateral sclerosis (ALS) is now known to represent a collection of overlapping syndromes. A better understanding of this heterogeneity and the ability to distinguish ALS subtypes would improve the clinical care of patients and enhance our understanding of the disease. Subtype profiles could be incorporated into the clinical trial design to improve our ability to detect a therapeutic effect. A variety of classification systems have been proposed over the years based on empirical observations, but it is unclear to what extent they genuinely reflect ALS population substructure. Methods We applied machine learning algorithms to a prospective, population-based cohort consisting of 2,858 Italian patients diagnosed with ALS for whom detailed clinical phenotype data were available. We replicated our findings in an independent population-based cohort of 1,097 Italian ALS patients. Findings We found that semi-supervised machine learning based on UMAP applied to the output of a multi-layered perceptron neural network produced the optimum clustering of the ALS patients in the discovery cohort. These clusters roughly corresponded to the six clinical subtypes defined by the Chiò classification system (bulbar ALS, respiratory ALS, flail arm ALS, classical ALS, pyramidal ALS, and flail leg ALS). The same clusters were identified in the replication cohort. A supervised learning approach based on ensemble learning identified twelve clinical parameters that predicted ALS clinical subtype with high accuracy (area under the curve = 0.94). Interpretation Our data-driven study provides insight into the ALS population's substructure and demonstrates that the Chiò classification system robustly identifies ALS subtypes. We provide an interactive website (https://share.streamlit.io/anant-dadu/machinelearningforals/main) so that clinical researchers can predict the clinical subtype of an ALS patient based on a small number of clinical parameters. Funding National Institute on Aging and the Italian Ministry of Health.

show abstract

“…Missingness is a common issue in EHR data with a potential impact on ML feature selection, model training, validation, and, importantly, interpretability. 5 Logistic regression falls under the umbrella of interpretable models. In other words, a human can understand the cause of the decisions.…”

mentioning

confidence: 99%

Linking Preoperative and Intraoperative Data for Risk Prediction

Mudumbai

Rashidi

2021

JAMA Netw Open

View full text Add to dashboard Cite

Elsewhere in JAMA Network Open, Xue and colleagues 1 share the results of their cohort study involving 111 888 surgeries at a large academic medical center. The research team deployed machine learning (ML) models to predict the risk of postoperative complications related to pneumonia, acute kidney injury, deep vein thrombosis, delirium, and pulmonary embolism. Studies indicate that more than 10% of surgical patients may experience a major postoperative complication (eg, heart attack, infection, and blood clots), with the incidence of postoperative complications varying by type of surgery. The use of ML approaches that take advantage of the rich storehouses of electronic health record (EHR) data and support perioperative clinical decision-making have been talked about for years; however, real-life examples are still few. Therefore, we applaud this group of investigators for pursuing this important and timely topic. Substantial evidence exists to support the prevention and early management of postoperative complications. The occurrence of a 30-day postoperative complication is more important than preoperative risk and intraoperative factors in determining survival after major surgery. For example, within 30 days of major surgery, between 14% and 30% of patients who develop a postoperative pulmonary complication (eg, pneumonia) will die vs 0.2% to 0.3% without the complication. 2

show abstract

Recommendations for Reporting Machine Learning Analyses in Clinical Research

Cited by 175 publications

References 32 publications

Public Covid-19 X-ray datasets and their impact on model bias - a systematic review of a significant problem

Public Covid-19 X-ray datasets and their impact on model bias - a systematic review of a significant problem

Identification and prediction of ALS subgroups using machine learning

Linking Preoperative and Intraoperative Data for Risk Prediction

Contact Info

Product

Resources

About