Prospective and External Evaluation of a Machine Learning Model to Predict In-Hospital Mortality of Adults at Time of Admission

Abstract

IMPORTANCE The ability to accurately predict in-hospital mortality for patients at the time of admission could improve clinical and operational decision-making and outcomes. Few of the machine learning models that have been developed to predict in-hospital death are both broadly applicable to all adult patients across a health system and readily implementable. Similarly, few have been implemented, and none have been evaluated prospectively and externally validated.OBJECTIVES To prospectively and externally validate a machine learning model that predicts in-hospital mortality for all adult patients at the time of hospital admission and to design the model using commonly available electronic health record data and accessible computational methods. DESIGN, SETTING, AND PARTICIPANTSIn this prognostic study, electronic health record data from a total of 43 180 hospitalizations representing 31 003 unique adult patients admitted to a quaternary academic hospital (hospital A) from October 1, 2014, to December 31, 2015, formed a training and validation cohort. The model was further validated in additional cohorts spanning from March 1, 2018, to August 31, 2018, using 16 122 hospitalizations representing 13 094 unique adult patients admitted to hospital A, 6586 hospitalizations representing 5613 unique adult patients admitted to hospital B, and 4086 hospitalizations representing 3428 unique adult patients admitted to hospital C. The model was integrated into the production electronic health record system and prospectively validated on a cohort of 5273 hospitalizations representing 4525 unique adult patients admitted to hospital A between February 14, 2019, and April 15, 2019. MAIN OUTCOMES AND MEASURES The main outcome was in-hospital mortality. Model performance was quantified using the area under the receiver operating characteristic curve and area under the precision recall curve. RESULTS A total of 75 247 hospital admissions (median [interquartile range] patient age, 59.5 [29.0]years; 45.9% involving male patients) were included in the study. The in-hospital mortality rates for the training validation; retrospective validations at hospitals A, B, and C; and prospective validation cohorts were 3.0%, 2.7%, 1.8%, 2.1%, and 1.6%, respectively. The area under the receiver operating characteristic curves were 0.87 (95% CI, 0.83-0.89), 0.85 (95% CI, 0.83-0.87), 0.89 (95% CI, 0.86-0.92), 0.84 (95% CI, 0.80-0.89), and 0.86 (95% CI, 0.83-0.90), respectively. The area under the precision recall curves were 0.29 (95% CI, 0.25-0.37), 0.17 (95% CI, 0.13-0.22), 0.22 (95% CI, 0.14-0.31), 0.13 (95% CI, 0.08-0.21), and 0.14 (95% CI, 0.09-0.21), respectively. CONCLUSIONS AND RELEVANCEProspective and multisite retrospective evaluations of a machine learning model demonstrated good discrimination of in-hospital mortality for adult patients at the (continued) Key Points Question How accurately can a machine learning model predict risk of in-hospital mortality for adult patients when evaluated prospectively and externally? Findings In this p...

A predictive model to estimate survival of hospitalized COVID-19 patients from admission data

“…In particular, during times of acute illness and hospitalization, ensuring appropriate consultations and care plans can be enabled by accurate models of survival and mortality. 8 Aiding healthcare workers with robust predictive survival models ensures more informed decision-making and efficient resource allocation while reducing physician stress and burnout. During the current pandemic, concerns about resource limitation and fair and appropriate allocation of resources 6 9 can be mitigated by clinically relevant, objective, and accurate decision-support prediction tools.…”

<p>Machine Learning Methods to Evaluate the Depression Status of Chinese Recruits: A Diagnostic Study</p>

“…50 The gradient enhancement model was better than the random forest and the regression in predicting mortality of patients admitted to the hospital. 51 The risk of suicide in American soldiers was evaluated using the Naive Bayes, random forest, SVM, and elastic net penalty regression, of which the sensitivity was optimal for an inelastic mesh classifier. 32 In another study of suicide risk in soldiers, the AUC of logistic regression was 0.62, while that of the Super Learner model was 0.83.…”

Machine learning to assist clinical decision-making during the COVID-19 pandemic

“…ML/AI models have been developed to be applied at each of these phases within ED care; in triage, vitals representing illness severity can predict resource needs (Levin et al 2018;Raita et al 2019) while risk profiles can be determined as new data is obtained (Janke et al 2016). Lastly, patient condition while in the ED can be used to calculate risk of deterioration or death (Brajer et al 2020). For patients with COVID-19, ML/AI can provide decision support at each stage by calculating likelihood of admission at triage, refining risk estimates with real data from clinical evaluation, and predicting the patient's trajectory as well as the effects of prompt ventilator use.…”