Multicenter Comparison of Machine Learning Methods and Conventional Regression for Predicting Clinical Deterioration on the Wards

Churpek, Matthew M.; Yuen, Trevor C.; Winslow, Christopher; Meltzer, David O.; Kattan, Michael W.; Edelson, Dana P.

doi:10.1097/ccm.0000000000001571

Cited by 490 publications

(487 citation statements)

References 34 publications

Supporting

Mentioning

465

Contrasting

Unclassified

Order By: Relevance

“…1 These moderately large AUROCs should not be surprising given the low event rate of CPA. Another study directly comparing classification modeling strategies for CPA (ie, logistic regression vs machine learning methods) has recently been published, 21 and the findings differed slightly from ours in that the random forest approach outperformed logistic regression with respect to AUROC (0.801 vs 0.770). The investigators also found that respiratory rate, heart rate, and age were the 3 most important predictor variables, whereas we found several laboratory values to be the most important clinical variables in our models.…”

Section: Discussioncontrasting

confidence: 74%

Advancing In-Hospital Clinical Deterioration Prediction Models

Jeffery¹,

Dietrich²,

Fabbri³

et al. 2018

Am J Crit Care

View full text Add to dashboard Cite

Background Early warning systems lack robust evidence that they improve patients’ outcomes, possibly because of their limitation of predicting binary rather than time-to-event outcomes. Objectives To compare the prediction accuracy of 2 statistical modeling strategies (logistic regression and Cox proportional hazards regression) and 2 machine learning strategies (random forest and random survival forest) for in-hospital cardiopulmonary arrest. Methods Retrospective cohort study with prediction model development from deidentified electronic health records at an urban academic medical center. Results The classification models (logistic regression and random forest) had statistical recall and precision similar to or greater than those of the time-to-event models (Cox proportional hazards regression and random survival forest). However, the time-to-event models provided predictions that could potentially better indicate to clinicians whether and when a patient is likely to experience cardiopulmonary arrest. Conclusions As early warning scoring systems are refined, they must use the best analytical methods that both model the underlying phenomenon and provide an understandable prediction. (American Journal of Critical Care. 2018; 27:381–391)

show abstract

Section: Discussioncontrasting

confidence: 74%

Advancing In-Hospital Clinical Deterioration Prediction Models

Jeffery¹,

Dietrich²,

Fabbri³

et al. 2018

Am J Crit Care

View full text Add to dashboard Cite

show abstract

“…Churpek et al confirmed that logistic regression and random forest outperformed a MEWS 32. Pirrachio et al developed the “Super ICU Learner Algorithm (SICULA)” using a combination of multiple machine learning methods for patients in intensive care units 44.…”

Section: Discussionmentioning

confidence: 99%

“…The details of SPTTS are shown in Table 3. In the previous studies, logistic regression and random forest were the most commonly used machine‐learning methods and showed better performance than traditional TTSs 32, 33. We used the area under the receiver operating characteristic curve (AUROC) and the area under the precision–recall curve (AUPRC) to measure the performance of the model.…”

Section: Methodsmentioning

confidence: 99%

An Algorithm Based on Deep Learning for Predicting In‐Hospital Cardiac Arrest

Kwon

Lee²,

Lee³

et al. 2018

JAHA

234

205

View full text Add to dashboard Cite

BackgroundIn‐hospital cardiac arrest is a major burden to public health, which affects patient safety. Although traditional track‐and‐trigger systems are used to predict cardiac arrest early, they have limitations, with low sensitivity and high false‐alarm rates. We propose a deep learning–based early warning system that shows higher performance than the existing track‐and‐trigger systems.Methods and ResultsThis retrospective cohort study reviewed patients who were admitted to 2 hospitals from June 2010 to July 2017. A total of 52 131 patients were included. Specifically, a recurrent neural network was trained using data from June 2010 to January 2017. The result was tested using the data from February to July 2017. The primary outcome was cardiac arrest, and the secondary outcome was death without attempted resuscitation. As comparative measures, we used the area under the receiver operating characteristic curve (AUROC), the area under the precision–recall curve (AUPRC), and the net reclassification index. Furthermore, we evaluated sensitivity while varying the number of alarms. The deep learning–based early warning system (AUROC: 0.850; AUPRC: 0.044) significantly outperformed a modified early warning score (AUROC: 0.603; AUPRC: 0.003), a random forest algorithm (AUROC: 0.780; AUPRC: 0.014), and logistic regression (AUROC: 0.613; AUPRC: 0.007). Furthermore, the deep learning–based early warning system reduced the number of alarms by 82.2%, 13.5%, and 42.1% compared with the modified early warning system, random forest, and logistic regression, respectively, at the same sensitivity.ConclusionsAn algorithm based on deep learning had high sensitivity and a low false‐alarm rate for detection of patients with cardiac arrest in the multicenter study.

show abstract

“…Early warning scores, such as the Modified Early Warning Score (MEWS) (17), can be used to risk stratify patients for additional surveillance. We have previously shown that an 11 PM MEWS score correlates well with overnight risk of clinical deterioration on the general wards (18) and that more advanced tools which incorporate vital signs, laboratory data and demographics with machine learning modeling are even more accurate (19). Further, these tools can be configured to automatically notify caregivers and even activate the rapid response team directly when patients pass a pre-set threshold.…”

Section: Discussionmentioning

confidence: 99%

Association Between Survival and Time of Day for Rapid Response Team Calls in a National Registry

Churpek

Edelson

Lee

et al. 2017

Critical Care Medicine

Self Cite

View full text Add to dashboard Cite

Objective Decreased staffing at nighttime is associated with worse outcomes in hospitalized patients. Rapid response teams (RRTs) were developed to decrease preventable harm by providing additional critical care resources to patients with clinical deterioration. We sought to determine whether RRT call frequency suffers from decreased utilization at night and how this is associated with patient outcomes. Design Retrospective analysis of a prospectively collected registry database Setting National registry database of inpatient RRT calls Patients Index RRT calls occurring on the general wards in the American Heart Association Get With the Guidelines-Medical Emergency Team database between 2005 and 2015 were analyzed. Interventions None Measurements and Main Results The primary outcome was in-hospital mortality. Patient and event characteristics between the hour with the highest and lowest mortality were compared, and multivariable models adjusting for patient characteristics were fit. A total of 282,710 RRT calls from 274 hospitals were included. The lowest frequency of calls occurred in the consecutive 1 AM–6:59 AM period, with 266/274 (97%) of hospitals having lower than expected call volumes during those hours. Mortality was highest during the 7 AM hour and lowest during the noon hour (18.8 vs. 13.8%; adjusted OR 1.41 (1.31–1.52); P<0.001). Compared to calls at the noon hour, those during the 7 AM hour had more deranged vital signs, were more likely to have a respiratory trigger, and were more likely to have >2 simultaneous triggers. Conclusions RRT activation is less frequent during the early morning and is followed by a spike in mortality in the 7 AM hour. These findings suggest that failure to rescue deteriorating patients is more common overnight. Strategies aimed at improving RRT utilization during these vulnerable hours may improve patient outcomes.

show abstract

Multicenter Comparison of Machine Learning Methods and Conventional Regression for Predicting Clinical Deterioration on the Wards

Cited by 490 publications

References 34 publications

Advancing In-Hospital Clinical Deterioration Prediction Models

Advancing In-Hospital Clinical Deterioration Prediction Models

An Algorithm Based on Deep Learning for Predicting In‐Hospital Cardiac Arrest

Association Between Survival and Time of Day for Rapid Response Team Calls in a National Registry

Contact Info

Product

Resources

About