Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU

Abstract

Background Patients in general medical-surgical wards who experience unplanned transfer to the intensive care unit (ICU) show evidence of physiologic derangement 6–24 h prior to their deterioration. With increasing availability of electronic medical records (EMRs), automated early warning scores (EWSs) are becoming feasible. Objective To describe the development and performance of an automated EWS based on EMR data. Materials and methods We used a discrete-time logistic regression model to obtain an hourly risk score to predict unplanned transfer to the ICU within the next 12 h. The model was based on hospitalization episodes from all adult patients (18 years) admitted to 21 Kaiser Permanente Northern California (KPNC) hospitals from 1/1/2010 to 12/31/2013. Eligible patients met these entry criteria: initial hospitalization occurred at a KPNC hospital; the hospitalization was not for childbirth; and the EMR had been operational at the hospital for at least 3 months. We evaluated the performance of this risk score, called Advanced Alert Monitor (AAM) and compared it against two other EWSs (eCART and NEWS) in terms of their sensitivity, specificity, negative predictive value, positive predictive value, and area under the receiver operator characteristic curve (c statistic). Results A total of 649,418 hospitalization episodes involving 374,838 patients met inclusion criteria, with 19,153 of the episodes experiencing at least one outcome. The analysis data set had 48,723,248 hourly observations. Predictors included physiologic data (laboratory tests and vital signs); neurological status; severity of illness and longitudinal comorbidity indices; care directives; and health services indicators (e.g. elapsed length of stay). AAM showed better performance compared to NEWS and eCART in all the metrics and prediction intervals. The AAM AUC was 0.82 compared to 0.79 and 0.76 for eCART and NEWS, respectively. Using a threshold that generated 1 alert per day in a unit with a patient census of 35, the sensitivity of AAM was 49% (95% CI: 47.6–50.3%) compared to the sensitivities of eCART and NEWS scores of 44% (42.3–45.1) and 40% (38.2–40.9), respectively. For all three scores, about half of alerts occurred within 12 h of the event, and almost two thirds within 24 h of the event. Conclusion The AAM score is an example of a score that takes advantage of multiple data streams now available in modern EMRs. It highlights the ability to harness complex algorithms to maximize signal extraction. The main challenge in the future is to develop detection approaches for patients in whom data are sparser because their baseline risk is lower.

A predictive model to estimate survival of hospitalized COVID-19 patients from admission data

“…Attempting to augment medical decision-making, studies ranging from modulating single parameters to advanced predictive modeling have been applied to forecast decompensation, mortality, and survival among other clinical outcomes. [24][25][26] Early work with small patient cohorts of COVID-19 has led to models that identify some clinical characteristics that can be applied to predict severe cases (Yan et al, 2020, Jiang et al, 2020. 27 28 However, these studies are limited to small numbers of patients as well as the inclusion of qualitative and subjective variables, are prone to mislabeling, and are not always readily available.…”

Enhanced early warning system impact on nursing practice: A phenomenological study

“…Early Warning Systems (EWS) are effective tools for identifying early signs of clinical deterioration (Mathukia, Fan, Vadyak, Biege, & Krishnamurthy, ; Umscheid et al., ). When automated and integrated into an electronic health record (EHR), these systems provide nurses with a real‐time score indicating the risk for deterioration and directing interventions based on the score (Kipnis et al., ; Umscheid et al., ). Critics of EWSs’ cite concerns that nurses will act on an EWS score instead of using clinical judgement to determine the significance of the score for a specific patient and to select the appropriate intervention (Alam et al., ; Bailey et al., ; McGaughey, O'Halloran, Porter, Trinder, & Blackwood, ).…”

Advancing In-Hospital Clinical Deterioration Prediction Models

“…24 The field of predictive analytics for in-hospital CPA continues to expand, as noted by people publishing prospective protocols 25 and testing additional statistical methods, such as discrete-time survival frameworks and generalized linear dynamic models. 26–28 We excluded patients who experienced CPA on their first day of care because we anticipated that different statistical strategies and model variables would be necessary to represent the phenomenon occurring earlier in a patient’s hospitalization (eg, using only emergency room triage data). Although we reported the heuristic advantage of noting trend line displays, we should investigate whether trends or point estimates are more likely to influence nurses’ behavior.…”