External validation of prognostic scores for COVID-19: a multicenter cohort study of patients hospitalized in Greater Paris University Hospitals

Abstract

Purpose The Coronavirus disease 2019 (COVID-19) has led to an unparalleled influx of patients. Prognostic scores could help optimizing healthcare delivery, but most of them have not been comprehensively validated. We aim to externally validate existing prognostic scores for COVID-19. Methods We used “COVID-19 Evidence Alerts” (McMaster University) to retrieve high-quality prognostic scores predicting death or intensive care unit (ICU) transfer from routinely collected data. We studied their accuracy in a retrospective multicenter cohort of adult patients hospitalized for COVID-19 from January 2020 to April 2021 in the Greater Paris University Hospitals. Areas under the receiver operating characteristic curves (AUC) were computed for the prediction of the original outcome, 30-day in-hospital mortality and the composite of 30-day in-hospital mortality or ICU transfer. Results We included 14,343 consecutive patients, 2583 (18%) died and 5067 (35%) died or were transferred to the ICU. We examined 274 studies and found 32 scores meeting the inclusion criteria: 19 had a significantly lower AUC in our cohort than in previously published validation studies for the original outcome; 25 performed better to predict in-hospital mortality than the composite of in-hospital mortality or ICU transfer; 7 had an AUC > 0.75 to predict in-hospital mortality; 2 had an AUC > 0.70 to predict the composite outcome. Conclusion Seven prognostic scores were fairly accurate to predict death in hospitalized COVID-19 patients. The 4C Mortality Score and the ABCS stand out because they performed as well in our cohort and their initial validation cohort, during the first epidemic wave and subsequent waves, and in younger and older patients. Supplementary Information The online version contains supplementary material available at 10.1007/s00134-021-06524-w.

Assess COVID-19 prognosis … but be aware of your instrument’s accuracy!

“…In conclusion, we strongly commend Lombardi et al [ 8 ] for their carefully conducted and comprehensive study. Although prior reports suggest that existing COVID-19 predictive models are at high risk for bias, this study helps lend confidence to a key set of prognostic scores by rigorously evaluating their performance in an independent sample.…”

“…In this issue of Intensive Care Medicine, Lombardi et al [ 8 ] evaluate the performance of existing prognostic scores to predict in-hospital mortality and the composite outcome of in-hospital mortality or ICU transfer in SARS-CoV-2-infected patients. The external validation was performed in a large database (14,343 patients) containing data collected in 39 hospitals spread across Paris and its surrounding region.…”

“…One of the major strengths of the study is that the external validation was performed using a large number of hospitals and patients, which strengthens the generalizability of the results. In addition, Lombardi et al [ 8 ] performed multiple supporting analyses to enhance the utility of their findings. First, they assessed discrimination using data from different waves of the pandemic (no substantial difference) and across age subgroups (lower performance was seen in patients > 65 years old for some scores).…”

Comparison of 5 acute pulmonary embolism mortality risk scores in patients with COVID-19

“…Thus, it is necessary to use both prognostic risk scores for PE and for COVID-19 mortality to better predict the prognosis in those patients. However as Lombardi Y [17] et al showed that the prognostic scores for COVID-19 were only fairly accurate to predict death in hospitalised COVID-19 patients.…”

Simple risk scores to predict hospitalization or death in outpatients with COVID-19 including the Omicron variant

“…However, while many COVID-19 risk scores have been proposed, almost all have been developed and validated to predict mortality in hospitalized patients. 1 These include the 4C risk score, 2 the ABCS risk score, 3 and the COVID-GRAM risk score. 4 Almost all of these risk scores require laboratory tests and in some cases imaging, which is often unavailable in the outpatient or telehealth settings.…”