Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models

Abstract

BACKGROUND: While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood. METHODS: We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed. RESULTS: Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73–0.78]) and validation (0.64 [interquartile range 0.60–0.67], P <0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59–0.90]) than distantly related cohorts (0.59 [interquartile range 0.43–0.73], P =0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation. CONCLUSIONS: There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.

ACCEPT 2·0: Recalibrating and externally validating the Acute COPD exacerbation prediction tool (ACCEPT)

“…In general, it is likely that prediction models of exacerbations need to be recalibrated to their specific settings (e.g., country, type of care [primary, secondary, or tertiary], and socioeconomic status), as has been demonstrated recently for cardiovascular risk prediction models. 39 Compared with exacerbation history alone, implementing ACCEPT requires knowledge of several variables and its calculations involves sophisticated non-linear regression model. This makes manual use of the model difficult.…”

Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

“…However, model performance can drift and models can lose their predictive abilities over time [7][8][9][10][11][12] . Drifts in the predictive performance of models can appear as a reduction in overall accuracy 13 or miscalibration 8,14,15 and can be due to changes in patient characteristics [14][15][16][17] , as well as new treatments, changes in preventative care, or changes in treatment algorithms 14 . These reductions in predictive performance of models can render these models less useful 18 or even misleading 18,19 , emphasizing the need to detect and correct these performance drifts.Various approaches to compensate for model performance calibration drift have been proposed in the past.…”

Do Prediction Models Do More Harm Than Good?

“…In this issue of Circulation: Cardiovascular Quality of Care and Outcomes , Gulati et al 2 present a rigorous thorough analysis of published clinical prediction models designed to guide clinical decision making in primary cardiovascular disease prevention, heart failure, and acute myocardial infarction. To do so, the authors tested the out-of-sample performance of the clinical prediction models in publicly available, external datasets.…”

“…The method used by Gulati et al 2 to evaluate these models is known as decision curve analysis, which depends on the concept of a threshold probability. 3 This concept will be familiar to cardiologists.…”

“…Gulati et al 2 demonstrate that many models produce inaccurate predictions that, if applied in clinical settings, could lead to worse outcomes than simply an amusing and misplaced advertisement or finding the wrong movie to stream. We must be prudent and vigilant when developing, testing, and implementing clinical prediction models, lest we harm the people we are trying to help.…”