Can we trust the prediction model? Demonstrating the importance of external validation by investigating the COVID-19 Vulnerability (C-19) Index across an international network of observational healthcare datasets

Abstract

Background: SARS-CoV-2 is straining healthcare systems globally. The burden on hospitals during the pandemic could be reduced by implementing prediction models that can discriminate between patients requiring hospitalization and those who do not. The COVID-19 vulnerability (C-19) index, a model that predicts which patients will be admitted to hospital for treatment of pneumonia or pneumonia proxies, has been developed and proposed as a valuable tool for decision making during the pandemic. However, the model is at high risk of bias according to the Prediction model Risk Of Bias ASsessment Tool and has not been externally validated. Methods: We followed the OHDSI framework for external validation to assess the reliability of the C-19 model. We evaluated the model on two different target populations: i) 41,381 patients that have SARS-CoV-2 at an outpatient or emergency room visit and ii) 9,429,285 patients that have influenza or related symptoms during an outpatient or emergency room visit, to predict their risk of hospitalization with pneumonia during the following 0 to 30 days. In total we validated the model across a network of 14 databases spanning the US, Europe, Australia and Asia. Findings: The internal validation performance of the C-19 index was a c-statistic of 0.73 and calibration was not reported by the authors. When we externally validated it by transporting it to SARS-CoV-2 data the model obtained c-statistics of 0.36, 0.53 (0.473-0.584) and 0.56 (0.488-0.636) on Spanish, US and South Korean datasets respectively. The calibration was poor with the model under-estimating risk. When validated on 12 datasets containing influenza patients across the OHDSI network the c-statistics ranged between 0.40-0.68. Interpretation: The results show that the discriminative performance of the C-19 model is low for influenza cohorts, and even worse amongst COVID-19 patients in the US, Spain and South Korea. These results suggest that C-19 should not be used to aid decision making during the COVID-19 pandemic. Our findings highlight the importance of performing external validation across a range of settings, especially when a prediction model is being extrapolated to a different population. In the field of prediction, extensive validation is required to create appropriate trust in a model.

Development and Validation of the Radiology Common Data Model (R-CDM) for the International Standardization of Medical Imaging Data

“… 22 As a response to the current coronavirus disease 2019 (COVID-19) pandemic, the OHDSI built distributed COVID-19 cohorts, evaluated a proposed algorithm to identify vulnerable patients, and developed a more reliable and reproducible algorithm for the same task using heterogeneous but standardized databases across the world. 23 24 25 As the proposed R-CDM is fully compatible with the clinical data of the OMOP-CDM, the implementation of the R-CDM can facilitate the development and evaluation of AI for radiology using standardized electronic phenotyping via collaborative and reproducible research. 26 …”

Integrating real-world data from Brazil and Pakistan into the OMOP common data model and standardized health analytics framework to characterize COVID-19 in the Global South

“…The OMOP CDM has been developed to work with a wide range of routinely collected health-care data; [14][15][16] numerous databases from North America, Europe, and beyond have been mapped to it. 24,25,28,29 The OMOP CDM has also been used to inform several studies relating to the COVID-19 pandemic. 17,[19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34] The FDN design allows for accelerated analytics with the same analysis code being run by each data partner and aggregated results shared, without any need to share patient-level data between data partners.…”

“…8 It has been leveraged to generate observational evidence for COVID-19 and has impacted international clinical guidelines and regulatory safety warnings. 17,[19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34] However, health data sources and data partners from low-and middle-income country (LMIC) settings remain largely underrepresented in such endeavors.…”

The Evolution of Clinical Knowledge During COVID-19: Towards a Global Learning Health System

“…The challenges faced in building sufficiently large data sets has meant that the modelling of COVID-19 has resulted in high risk of bias and poor external validation [ 51 52 53 ]. Additionally, the inherent nature of observational EHR-based studies, lacking controlled cohort selection, may lead to unreliable results due to confounding [ 5 ] and a risk of case contamination due to ambiguous cohort definitions [ 54 ].…”