Real-Time Electronic Health Record Mortality Prediction During the COVID-19 Pandemic: A Prospective Cohort Study

Abstract

BackgroundThe SARS-CoV-2 virus has infected millions of people, overwhelming critical care resources in some regions. Many plans for rationing critical care resources during crises are based on the Sequential Organ Failure Assessment (SOFA) score. The COVID-19 pandemic created an emergent need to develop and validate a novel electronic health record (EHR)-computable tool to predict mortality.Research QuestionsTo rapidly develop, validate, and implement a novel real-time mortality score for the COVID-19 pandemic that improves upon SOFA.Study Design and MethodsWe conducted a prospective cohort study of a regional health system with 12 hospitals in Colorado between March 2020 and July 2020. All patients >14 years old hospitalized during the study period without a do not resuscitate order were included. Patients were stratified by the diagnosis of COVID-19. From this cohort, we developed and validated a model using stacked generalization to predict mortality using data widely available in the EHR by combining five previously validated scores and additional novel variables reported to be associated with COVID-19-specific mortality. We compared the area under the receiver operator curve (AUROC) for the new model to the SOFA score and the Charlson Comorbidity Index.ResultsWe prospectively analyzed 27,296 encounters, of which 1,358 (5.0%) were positive for SARS-CoV-2, 4,494 (16.5%) included intensive care unit (ICU)-level care, 1,480 (5.4%) included invasive mechanical ventilation, and 717 (2.6%) ended in death. The Charlson Comorbidity Index and SOFA scores predicted overall mortality with an AUROC of 0.72 and 0.90, respectively. Our novel score predicted overall mortality with AUROC 0.94. In the subset of patients with COVID-19, we predicted mortality with AUROC 0.90, whereas SOFA had AUROC of 0.85.InterpretationWe developed and validated an accurate, in-hospital mortality prediction score in a live EHR for automatic and continuous calculation using a novel model, that improved upon SOFA.Take Home PointsStudy QuestionCan we improve upon the SOFA score for real-time mortality prediction during the COVID-19 pandemic by leveraging electronic health record (EHR) data?ResultsWe rapidly developed and implemented a novel yet SOFA-anchored mortality model across 12 hospitals and conducted a prospective cohort study of 27,296 adult hospitalizations, 1,358 (5.0%) of which were positive for SARS-CoV-2. The Charlson Comorbidity Index and SOFA scores predicted all-cause mortality with AUROCs of 0.72 and 0.90, respectively. Our novel score predicted mortality with AUROC 0.94.InterpretationA novel EHR-based mortality score can be rapidly implemented to better predict patient outcomes during an evolving pandemic.

Clinical Characterization and Prediction of Clinical Severity of SARS-CoV-2 Infection Among US Adults Using Data From the US National COVID Cohort Collaborative

“…However, development of such tools would require additional work at deploying health care systems, including user engagement, workflow analysis, variable mapping and internal validation, and consideration of any desired visualizations and alerts. 35 …”

Predictive Value of Sequential Organ Failure Assessment Score across Patients with and without COVID-19 Infection

“…While not previously compared head-to-head, this accuracy is lower than the outstanding accuracy found by Sottile et al (AUROC = 0.90) in a cohort of hospitalizations inclusive of patients both with and without COVID-19. 8 In this latter study, however, the SOFA scores evaluated were the maximum at any point during hospitalization and not just within the first 48 hours following admission. It is not surprising that SOFA defined in this manner has enhanced accuracy as it incorporates more data and may include high scores occurring just prior to deaths.…”

“…2 Our findings of excellent SOFA accuracy (AUROC 0.835) among COVID-19 positive hospitalizations are consistent with those of Sottile et al using data from Colorado (AUROC 0.85). 8 Ma et al noted a substantially poor accuracy for COVID-19 hospitalizations, 5 yet their cohort was based in China and included patients from early on in the pandemic whose treatment and outcomes likely differed from those infected with COVID-19 more recently.…”

“…6,7 In the US, SOFA scores had excellent predictive accuracy for hospitalized COVID-19 patients (AUROC 0.85), and performed even better in a larger cohort inclusive of non-COVID-19 patients (AUROC 0.90). 8 Whether the accuracy of SOFA differs for patients based on COVID-19 status is unknown.…”

Characteristics Associated With Disparities Among Older Adults in Coronavirus Disease 2019 Outcomes in an Academic Health Care System

“…We also used ICD-10 codes to identify serious illnesses and leading causes of death in the United States, including cancer, COPD, dementia, and coronary artery disease (CAD) 20,21. To capture the severity of illness, we included the worst Sequential Organ Failure Assessment (SOFA) Score in the first 7 days of presentation 22–24. We included receipt of treatments during the ED/hospital encounter, such as dexamethasone,25 the mutually exclusive maximal oxygen support received (nasal cannula, high-flow nasal cannula, bilevel positive airway pressure, or ventilator), and ICU admission.…”