Reporting of Model Performance and Statistical Methods in Studies That Use Machine Learning to Develop Clinical Prediction Models: Protocol for a Systematic Review

Abstract

Background With the growing excitement of the potential benefits of using machine learning and artificial intelligence in medicine, the number of published clinical prediction models that use these approaches has increased. However, there is evidence (albeit limited) that suggests that the reporting of machine learning–specific aspects in these studies is poor. Further, there are no reviews assessing the reporting quality or broadly accepted reporting guidelines for these aspects. Objective This paper presents the protocol for a systematic review that will assess the reporting quality of machine learning–specific aspects in studies that use machine learning to develop clinical prediction models. Methods We will include studies that use a supervised machine learning algorithm to develop a prediction model for use in clinical practice (ie, for diagnosis or prognosis of a condition or identification of candidates for health care interventions). We will search MEDLINE for studies published in 2019, pseudorandomly sort the records, and screen until we obtain 100 studies that meet our inclusion criteria. We will assess reporting quality with a novel checklist developed in parallel with this review, which includes content derived from existing reporting guidelines, textbooks, and consultations with experts. The checklist will cover 4 key areas where the reporting of machine learning studies is unique: modelling steps (order and data used for each step), model performance (eg, reporting the performance of each model compared), statistical methods (eg, describing the tuning approach), and presentation of models (eg, specifying the predictors that contributed to the final model). Results We completed data analysis in August 2021 and are writing the manuscript. We expect to submit the results to a peer-reviewed journal in early 2022. Conclusions This review will contribute to more standardized and complete reporting in the field by identifying areas where reporting is poor and can be improved. Trial Registration PROSPERO International Prospective Register of Systematic Reviews CRD42020206167; https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=206167 International Registered Report Identifier (IRRID) RR1-10.2196/30956

Reducing Avoidable Emergency Visits and Hospitalizations With Patient Risk-Based Prescriptive Analytics: A Quality Improvement Project at an Oncology Care Model Practice

“…Patient risk-based descriptive analytics using ML will become increasingly prominent as new practice reimbursement schemes such as the Enhancing Oncology Model (EOM) replacing OCM are implemented, and owing to their flexibility and inherently complex nature, proper reporting will be key to realizing their potential and attenuating misuse. 21,22 The AI tool has enabled nurse case managers to identify and resolve critical clinical issues and reduce avoidable ACU. Effects on outcomes can be inferred from the reduction; targeting short-term interventions toward patients most at-risk translates to better long-term care and outcomes.…”

“…Patient risk-based descriptive analytics using ML will become increasingly prominent as new practice reimbursement schemes such as the Enhancing Oncology Model (EOM) replacing OCM are implemented, and owing to their flexibility and inherently complex nature, proper reporting will be key to realizing their potential and attenuating misuse. 21 , 22 …”

Método de random forest para el reconocimiento de patrones de sensibilidad y resistencia en antibiogramas

“…Los métodos analíticos ofrecen mayor desempeño para identificar variables dependientes y niveles de correspondencia (Pearson, Spearman Rank) al igual que los modelos de regresión lineal simple y múltiple. Sin embargo, son sensibles a la varianza de valores extremos y no tienen la capacidad de clasificar patrones en variables categóricas en miles de millones de datos 1 .…”

Credit Card Default Prediction with Data Modeling

“…Even though random forest and XGBoost have around 90% accuracy, the precision is not good enough. There are still enough false positives which may cause the loss of potential customers, and the false negatives can cause massive business losses [3].…”