Predictors for independent external validation of cardiovascular risk clinical prediction rules: Cox proportional hazards regression analyses

Abstract

Background: Clinical prediction rules (CPRs) should be externally validated by independent researchers. Although there are many cardiovascular CPRs, most have not been externally validated. It is not known why some CPRs are externally validated by independent researchers and others are not. Methods: We analyzed cardiovascular risk CPRs included in a systematic review. Independent external validations were identified by forward citation searches of derivation studies. Time between the publication of a cardiovascular CPR and the first independent external validation was calculated. We assessed Kaplan-Meier estimates of the probability to have an independent external validation. Using univariable Cox regression, we explored whether characteristics of derivation (design, location, sample size, number of predictors, presentation format, validation in derivation), reporting (participants, predictors, outcomes, performance measure, information for risk calculation), and publication (journal impact factor) are associated with time to the first independent external validation. Results: Of 125 cardiovascular risk CPRs, 29 had an independent external validation. The median follow-up was 118 months (95% CI, 99-130). The 25th percentile of event time was 122 months (95% CI, 91-299). Cardiovascular risk CPRs from the USA were 4.15 times (95% CI, 1.89-9.13) more likely to have an independent external validation. Increasing the sample size of derivation by ten times was associated with a 2.32-fold (95% CI, 1.37-3.91) increase in the probability of having an independent external validation. CPRs presented with an internal validation tend to get an independent external validation sooner (HR = 1.73, 95% CI, 0.77-3.93). CPRs reporting all the information necessary for calculating individual risk were 2.65 (95% CI, 1.01-6.96) times more likely to have an independent external validation. Publishing a cardiovascular risk CPR in a journal that has one unit higher impact factor was associated with a 6% (95% CI, 3-9) higher likelihood of an independent external validation. Conclusions: The probability for cardiovascular risk CPRs to get an independent external validation was low even many years after their derivations. Authors of new cardiovascular risk CPRs should consider using adequate sample size, conducting an internal validation, and reporting all the information needed for risk calculation as these features were associated with an independent external validation.

A prognostic model for overall survival in sporadic Creutzfeldt‐Jakob disease

“…The selected prognostic model was calibrated well (Figure A.1 in supporting information) and the observed median disease duration was well captured by the predicted median disease duration (Figure A.2 in supporting information). The baseline survival (information that is necessary for calculating individual risk 28 ) was 99.6% at 1 month. A score chart was derived to predict 6‐month survival and median disease duration based on the selected model (Figure 3).…”

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

“…Other types of external validation studies include methodologic validation which refers to testing using data collected via different methods, spectrum validation which refers to testing in patients with different disease severity or prevalence of the outcome of interest and fully independent validation which refers to testing by independent investigators at different sites [26, 147]. A recent study of cardiovascular risk CPRs found that very few were externally validated by independent researchers; to increase the chance of fully independent validation, researchers should report all the information required for risk calculation, to ensure replicability [178]. Some authors have found that CPRs demonstrate worse performance in fully independent external validation studies compared to temporal or geographical external validation studies [26, 28], while others have found no difference [179].…”

Re-estimation improved the performance of two Framingham cardiovascular risk equations and the Pooled Cohort equations: A nationwide registry analysis

“…More updating studies are required. External validation studies evaluating the transportability of a risk equation to a new population are still not common; e.g., in a recent systematic review, Ban et al found that only 23% of 125 CVD equations were externally validated 32 . Updating studies combining evidence from published equations with new data to tailor the prediction tool to other populations or to update them because of changes over time are even rarer.…”