A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice*

Abstract

Objectives: Develop and implement a machine learning algorithm to predict severe sepsis and septic shock and evaluate the impact on clinical practice and patient outcomes. Design: Retrospective cohort for algorithm derivation and validation, pre-post impact evaluation. Setting: Tertiary teaching hospital system in Philadelphia, PA. Patients: All non-ICU admissions; algorithm derivation July 2011 to June 2014 (n = 162,212); algorithm validation October to December 2015 (n = 10,448); silent versus alert comparison January 2016 to February 2017 (silent n = 22,280; alert n = 32,184). Interventions: A random-forest classifier, derived and validated using electronic health record data, was deployed both silently and later with an alert to notify clinical teams of sepsis prediction. Measurement and Main Result: Patients identified for training the algorithm were required to have International Classification of Diseases, 9th Edition codes for severe sepsis or septic shock and a positive blood culture during their hospital encounter with either a lactate greater than 2.2 mmol/L or a systolic blood pressure less than 90 mm Hg. The algorithm demonstrated a sensitivity of 26% and specificity of 98%, with a positive predictive value of 29% and positive likelihood ratio of 13. The alert resulted in a small statistically significant increase in lactate testing and IV fluid administration. There was no significant difference in mortality, discharge disposition, or transfer to ICU, although there was a reduction in time-to-ICU transfer. Conclusions: Our machine learning algorithm can predict, with low sensitivity but high specificity, the impending occurrence of severe sepsis and septic shock. Algorithm-generated predictive alerts modestly impacted clinical measures. Next steps include describing clinical perception of this tool and optimizing algorithm design and delivery.

Effect of a sepsis prediction algorithm on patient mortality, length of stay and readmission: a prospective multicentre clinical outcomes evaluation of real-world patient data from US hospitals

“…28 Beyond the simple heuristics of rules-based scoring systems such as MEWS, SOFA, qSOFA and SIRS, several machine learning approaches have been retrospectively evaluated for the detection and prediction of incipient sepsis. 37 38 57-66 They include dynamic Bayesian networks, 60 support vector machines, 57 survival-analytical models (TREWScore, Artificial Intelligence Sepsis Expert), 61 62 smoothed disease severity score learning, 63 hierarchical switching linear dynamical systems, 64 autoregressive hidden Markov models, 65 free-text models 38 and random-forest models. 57 These tools contribute notably to the field of sepsis detection because they offer generalisability, are scalable, and can be updated as new information is acquired.…”

“…37 38 57-66 They include dynamic Bayesian networks, 60 support vector machines, 57 survival-analytical models (TREWScore, Artificial Intelligence Sepsis Expert), 61 62 smoothed disease severity score learning, 63 hierarchical switching linear dynamical systems, 64 autoregressive hidden Markov models, 65 free-text models 38 and random-forest models. 57 These tools contribute notably to the field of sepsis detection because they offer generalisability, are scalable, and can be updated as new information is acquired. 58 However, many do not use information about measurement trends or correlations, 67 or do so ineffectively.…”

Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy

“…Interventional studies using traditional SIRS and SOFA alarm systems have not shown significant changes in clinical outcomes [48][49][50]. Only three interventional studies have been identified in this review, which were carried out in different clinical settings and show mixed results [31,32,51]. None of the studies, however, investigated a direct clinical action associated with the sepsis prediction, but left treatment decisions at the discretion of the clinician.…”

“…One clinical validation study in the ED showed the machine learning model outperformed manual scoring by nurses and the SIRS criteria when identifying severe sepsis and septic shock [28], the other study made no comparison [29]. The interventional studies included two pre-post implementation studies (in-hospital) [30,31] and one ICU randomized controlled trial [32]. All looked at mortality and hospital length of stay, but results are mixed as shown in Table 1.…”

Practical Machine Learning-Based Sepsis Prediction

“…Previous studies explored the potential for ML-based approaches to enhance ICU patient outcomes with sepsis [9]; however, the effectiveness and utility of these algorithms on clinical practice and patient outcomes, particularly outside of ICU settings, are yet to be established [10]. It is important to consider how sepsis prediction changes with the data available in these different settings, how patients need to be monitored across settings, as well as the multitude of factors that can influence the performance of sepsis prediction in diverse populations.…”