Validation of a Derived International Patient Severity Algorithm to Support COVID-19 Analytics from Electronic Health Record Data

Abstract

Introduction. The Consortium for Clinical Characterization of COVID-19 by EHR (4CE) includes hundreds of hospitals internationally using a federated computational approach to COVID-19 research using the EHR. Objective. We sought to develop and validate a standard definition of COVID-19 severity from readily accessible EHR data across the Consortium. Methods. We developed an EHR-based severity algorithm and validated it on patient hospitalization data from 12 4CE clinical sites against the outcomes of ICU admission and/or death. We also used a machine learning approach to compare selected predictors of severity to the 4CE algorithm at one site. Results. The 4CE severity algorithm performed with pooled sensitivity of 0.73 and specificity 0.83 for the combined outcome of ICU admission and/or death. The sensitivity of single code categories for acuity were unacceptably inaccurate - varying by up to 0.65 across sites. A multivariate machine learning approach identified codes resulting in mean AUC 0.956 (95% CI: 0.952, 0.959) compared to 0.903 (95% CI: 0.886, 0.921) using expert-derived codes. Billing codes were poor proxies of ICU admission, with 49% precision and recall compared against chart review at one partner institution. Discussion. We developed a proxy measure of severity that proved resilient to coding variability internationally by using a set of 6 code classes. In contrast, machine-learning approaches may tend to overfit hospital-specific orders. Manual chart review revealed discrepancies even in the gold standard outcomes, possibly due to pandemic conditions. Conclusion. We developed an EHR-based algorithm for COVID-19 severity and validated it at 12 international sites.

Development of a COVID-19 Application Ontology for the ACT Network

“…Table 1 lists the emerging codes that are included in the ontology. 2) We created computable phenotypes to characterize the course of illness and outcomes in COVID-19 that included illness severity [12], respiratory therapy management, and level of care. We developed computable phenotypes for three levels of illness severity – moderate, severe, and death – and for four levels of respiratory therapy management – supplemental oxygen, intubation, mechanical ventilation, and extracorporeal membrane oxygenation (ECMO) – and for each of these phenotypes we collected a set of relevant codes from ICD-10, CPT-4, and DRG.…”

“…Though we primarily developed the COVID-19 ontology for the ACT network, the ontology is has been leveraged by other COVID-19 research groups. In particular, the computable phenotypes that characterize the severity COVID-19 and disease outcomes have be beneficial in standardizing these computable phenotype definitions beyond the ACT network [12]. Since the publication of the ontology, the 4CE, the N3C, and the PCORnet consortia have adapted the ontology for their own use.…”

Multinational Prevalence of Neurological Phenotypes in Patients Hospitalized with COVID-19

“…1 ) as well as COVID-19 severity, defined according to the 4CE definition. 24 Of the 45 contributing sites, five sites (all from Italy) provided only ICD-9 codes while the remaining 40 sites provided predominantly ICD-10 codes. As such, we used ICD-10 data for the main analyses and ICD-9 data for supplementary analyses.…”

“…It could take a few days from SARS-CoV-2 infection to symptom onset and additional days before a positive PCR test and/or hospital admission. Similarly, we analyzed ICD codes before and after the admission date according to whether patients ever met the 4CE criteria for severe COVID-19 24 ( Fig. 1 ).…”

“…We first examined all hospitalized patients with positive SARS-CoV-2 PCR tests. We then examined hospitalized patients with COVID-19 who met any of the definitions for severe COVID-19 24 , including advanced respiratory care management at any point during their hospitalization. Definitions of severe COVID-19 included diagnoses, procedures, laboratory results, or medications ( Table 1 ).…”