A Simulated Prospective Evaluation of a Deep Learning Model for Real-Time Prediction of Clinical Deterioration Among Ward Patients*

Shah, Parth; Ginestra, Jennifer C; Ungar, Lyle H.; Junker, Paul; Rohrbach, Jeffrey; Fishman, Neil O.; Weissman, Gary E.

doi:10.1097/ccm.0000000000004966

Cited by 11 publications

(12 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…13,28,29 Our model also performed significantly better than a deep recurrent neural network trained on a similar set of features including hourly vitals. 30 These findings suggest that random forest may be the "algorithm of choice" for real-time deterioration scores.…”

Section: Discussionmentioning

confidence: 99%

Real-time Machine Learning Alerts to Prevent Escalation of Care: A Pragmatic Clinical Trial

Levin¹,

Kia²,

Timsina³

et al. 2022

Preprint

View full text Add to dashboard Cite

Importance: Automated machine learning algorithms have been shown to outperform older methods in predicting clinical deterioration requiring escalation of care, but rigorous prospective data on their real-world efficacy are limited. Objective: We hypothesized that real-time deterioration prediction alerts sent directly to front-line providers would reduce escalations. Design: Single-center prospective pragmatic clinical trial conducted from July 2019 to March 2020. The trial was terminated early due to the COVID-19 pandemic. Patients were followed for 30 days post-discharge. Setting: Academic tertiary care medical center located in New York City. Participants: All adult patients admitted to any of four medical-surgical units were included. Assignment to intervention or control arms was determined by initial unit admission. Intervention: Real-time alerts stratified according to predicted likelihood of clinical deterioration sent to the nursing/primary team or directly to the rapid response team. Clinical care and interventions were at the discretion of the providers. For the control units, alerts were generated but not sent. Main Outcomes: The primary outcome was the incidence of escalation of care. Secondary outcomes included orders placed for cardiovascular support, in-hospital and 30-day mortality. Ad-hoc outcomes included time to ICU escalation and time to discharge. Results: 2,780 patients were enrolled, 1,506 in the intervention group and the 1,274 in the control group. Average age was 66.2 years and 1,446 (52%) of participants were female. There was no difference in escalation between the groups, relative risk(RR) 1.22(95% Confidence Interval[CI] (0.97,1.54),p=0.10). Patients in the intervention group were more likely to receive cardiovascular support orders RR 1.35(95% CI (1.10,1.66),p=0.022). Median time to escalation with alerts was 50.6 [21.6-103] versus 58.6 [25.4-115] hours (difference -5.70;95% CI (-10.00,-2.00),p<0.001). The hazard ratio for likelihood of ICU escalation within 12 hours of an alert was 3.36 (95% CI (1.38,8.21),p=0.01) and for faster hospital discharge was 1.10 (95% CI (1.01,1.19),p=0.02). Combined in-hospital and 30-day-mortality was lower in the intervention group, RR 0.72 (95% CI (0.54,0.94),p=0.01). Conclusions and Relevance: Preliminary evidence suggests that real-time machine learning alerts do not reduce the incidence of escalation but are effective in reducing time to escalation, hospital length of stay and mortality. Trial Registration: ClinicalTrials.gov,NCT04026555, https://clinicaltrials.gov

show abstract

Section: Discussionmentioning

confidence: 99%

Real-time Machine Learning Alerts to Prevent Escalation of Care: A Pragmatic Clinical Trial

Levin¹,

Kia²,

Timsina³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Artificial intelligence (AI)‐based predictive analytics provide new opportunities to leverage rich sources of continuous data to improve patient care 1‐8 . These approaches use machine learning and other modern statistical techniques to provide early warning of events of clinical deterioration such as sepsis, respiratory failure, hemorrhage, and emergent intensive care unit (ICU) transfer 9‐11 .…”

Section: Introductionmentioning

confidence: 99%

“…Artificial intelligence (AI)‐based predictive analytics provide new opportunities to leverage rich sources of continuous data to improve patient care. 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 These approaches use machine learning and other modern statistical techniques to provide early warning of events of clinical deterioration such as sepsis, respiratory failure, hemorrhage, and emergent intensive care unit (ICU) transfer. 9 , 10 , 11 While AI‐based predictive analytics estimate the risk of specific clinical events, in practice they can be used as a proxy for illness severity, or even as a comprehensive biomarker or physiomarker.…”

Section: Introductionmentioning

confidence: 99%

Beyond prediction: Off‐target uses of artificial intelligence‐based predictive analytics in a learning health system

Keim‐Malpass

Moorman²,

Monfredi

et al. 2022

Learning Health Systems

View full text Add to dashboard Cite

Introduction: Artificial-intelligence (AI)-based predictive analytics provide new opportunities to leverage rich sources of continuous data to improve patient care through early warning of the risk of clinical deterioration and improved situational awareness.Part of the success of predictive analytic implementation relies on integration of the analytic within complex clinical workflows. Pharmaceutical interventions have offtarget uses where a drug indication has not been formally studied for a different indication but has potential for clinical benefit. An analog has not been described in the context of AI-based predictive analytics, that is, when a predictive analytic has been trained on one outcome of interest but is used for additional applications in clinical practice. Methods:In this manuscript we present three clinical vignettes describing off-target use of AI-based predictive analytics that evolved organically through real-world practice.Results: Off-target uses included:real-time feedback about treatment effectiveness, indication of readiness to discharge, and indication of the acuity of a hospital unit. Conclusion:Such practice fits well with the learning health system goals to continuously integrate data and experience to provide.

show abstract

“…Because general ward patients are observed less closely than in the ED or ICU setting with fewer vital signs documented and laboratory tests performed, they represent a proportionally more vulnerable population that could benefit more from an augmented sepsis early warning system. Therefore, the objective of this study was to develop a machine learning model for predicting sepsis in the general ward setting, compare its performance to commonly used instruments for sepsis surveillance such as SIRS and NEWS, and extend the model evaluation using a novel simulated pseudo-prospective trial ( 13 , 14 ).…”

Section: Introductionmentioning

confidence: 99%

“…of the Sepsis-3 criteria identified 2,206 (3.1%) patient with sepsis encounters. Patients with sepsis were slightly older [65.6 (56.3-74.3) vs. 60.8 (49.4-71.2), p < 0.01], more likely to be white (71.3 vs. 61.8%, p < 0.01), had a higher Elixhauser comorbidity score [19(10)(11)(12)(13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29) vs. 9 (1-17), p < 0.01], a longer length of stay [12.9 (8.0-19.3) vs. 3.9 (2.3-6.7), p < 0.01], and higher inpatient mortality (16.6% vs. 0.8%, p < 0.01) (…”

mentioning

confidence: 99%

Sepsis Prediction for the General Ward Setting

Gupta

Betthauser

et al. 2022

Front. Digit. Health

View full text Add to dashboard Cite

ObjectiveTo develop and evaluate a sepsis prediction model for the general ward setting and extend the evaluation through a novel pseudo-prospective trial design.DesignRetrospective analysis of data extracted from electronic health records (EHR).SettingSingle, tertiary-care academic medical center in St. Louis, MO, USA.PatientsAdult, non-surgical inpatients admitted between January 1, 2012 and June 1, 2019.InterventionsNone.Measurements and Main ResultsOf the 70,034 included patient encounters, 3.1% were septic based on the Sepsis-3 criteria. Features were generated from the EHR data and were used to develop a machine learning model to predict sepsis 6-h ahead of onset. The best performing model had an Area Under the Receiver Operating Characteristic curve (AUROC or c-statistic) of 0.862 ± 0.011 and Area Under the Precision-Recall Curve (AUPRC) of 0.294 ± 0.021 compared to that of Logistic Regression (0.857 ± 0.008 and 0.256 ± 0.024) and NEWS 2 (0.699 ± 0.012 and 0.092 ± 0.009). In the pseudo-prospective trial, 388 (69.7%) septic patients were alerted on with a specificity of 81.4%. Within 24 h of crossing the alert threshold, 20.9% had a sepsis-related event occur.ConclusionsA machine learning model capable of predicting sepsis in the general ward setting was developed using the EHR data. The pseudo-prospective trial provided a more realistic estimation of implemented performance and demonstrated a 29.1% Positive Predictive Value (PPV) for sepsis-related intervention or outcome within 48 h.

show abstract

A Simulated Prospective Evaluation of a Deep Learning Model for Real-Time Prediction of Clinical Deterioration Among Ward Patients*

Cited by 11 publications

References 35 publications

Real-time Machine Learning Alerts to Prevent Escalation of Care: A Pragmatic Clinical Trial

Real-time Machine Learning Alerts to Prevent Escalation of Care: A Pragmatic Clinical Trial

Beyond prediction: Off‐target uses of artificial intelligence‐based predictive analytics in a learning health system

Sepsis Prediction for the General Ward Setting

Contact Info

Product

Resources

About