Early Warning Systems: The Neglected Importance of Timing

Abstract

A utomated early warning systems (EWSs) use data inputs to recognize clinical states requiring time-sensitive intervention and then generate notifications through different modalities to clinicians. EWSs serve as common tools for improving the recognition and treatment of important clinical states such as sepsis. However, despite the early enthusiasm, these warning systems have often yielded disappointing outcomes. In sepsis, for example, EWSs have shown mixed results in clinical trials, and concerns regarding the overuse of EWSs in diagnosing sepsis have grown. [1][2][3][4] We argue that inattention to the importance of timing in EWS training and evaluation provides one reason that EWSs have underperformed. Thus, to improve care, a warning system must not only identify the clinical state accurately, but it must also do so in a sufficiently timely manner to implement the associated interventions, such as administration of antibiotics for sepsis. Although the literature has occasionally highlighted the importance of timing in electronic surveillance systems, no one has linked the temporal dependence of performance metrics and intervention feasibility to the failure of such warning systems and explained how to operationalize timing in their development. 5-8 Using sepsis as an example, we explain why timing is important and propose new metrics and strategies for training and evaluating EWS models. EWSs are divided into two types: detection systems that recognize critical illnesses at a particular moment and prediction systems that estimate risk of deterioration over varying time frames. 9 We focus primarily on detection systems, but our analysis is also important for prediction systems, which we will discuss in the last section. CLINICAL TIME ZERO AND POSITIVE PREDICTIVE VALUEEWS metrics have evolved from focusing on crude measures of discrimination to more clinically relevant metrics, such as the positive predictive value (PPV). The common performance metrics, including the c-statistic, evaluate the performance of EWSs in distinguishing events from nonevents, such as the presence or absence of sepsis in hospitalized patients. How-ever, the c-statistic does not account for disease prevalence. A given c-statistic is compatible with a wide range of PPVs; a low PPV may limit an EWS's usefulness to promote interventions and generate increased alert fatigue. 10 However, the PPV, although important, provides no information on the timing of state recognition in relation to clinical time zero. Time zero is the first moment at which a critical state can be recognized based on available data and current medical science. Different approaches, including laboratory values, clinical assessments, retrospective chart reviews, triage times, and others, have been used to measure time zero. 8,[11][12][13] All these approaches feature advantages and disadvantages; the evaluation of timing will exhibit sensitivity to the approach used. 14 Further work is needed to gain additional insights into the measurement of time zero.Just as the sa...

External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients

“…Many models have been developed to improve timely identification of sepsis, but their lack of adoption has led to an implementation gap in early warning systems for sepsis . This gap has largely been filled by commercial electronic health record (EHR) vendors, who have integrated early warning systems into the EHR where they can be readily accessed by clinicians and linked to clinical interventions . More than half of surveyed US health systems report using electronic alerts, with nearly all using an alert system for sepsis …”

Preferences for Predictive Model Characteristics among People Living with Chronic Lung Disease: A Discrete Choice Experiment

“…8 The choice of the predictive time horizon should be long enough to allow for some intervention or change in behavior while being short enough to feel relevant to a decision in the present. 9…”

Quality and safety in the literature: October 2022

“…However, lack of widespread adoption of these models has led to a vacancy of early warning systems that has been largely filled by electronic health record (EHR) vendors. In fact, in 2016, more than half of US health systems reported use of electronic alerts, nearly all of which include an alert system for sepsis 4 5. The reviewed study evaluated one of the most widely used early warning systems in the USA: the Epic Sepsis Model (ESM), a proprietary model that had yet to be externally validated 6…”