Use and Customization of Risk Scores for Predicting Cardiovascular Events Using Electronic Health Record Data

Abstract

BackgroundClinicians who are using the Framingham Risk Score (FRS) or the American College of Cardiology/American Heart Association Pooled Cohort Equations (PCE) to estimate risk for their patients based on electronic health data (EHD) face 4 questions. (1) Do published risk scores applied to EHD yield accurate estimates of cardiovascular risk? (2) Are FRS risk estimates, which are based on data that are up to 45 years old, valid for a contemporary patient population seeking routine care? (3) Do the PCE make the FRS obsolete? (4) Does refitting the risk score using EHD improve the accuracy of risk estimates?Methods and ResultsData were extracted from the EHD of 84 116 adults aged 40 to 79 years who received care at a large healthcare delivery and insurance organization between 2001 and 2011. We assessed calibration and discrimination for 4 risk scores: published versions of FRS and PCE and versions obtained by refitting models using a subset of the available EHD. The published FRS was well calibrated (calibration statistic K=9.1, miscalibration ranging from 0% to 17% across risk groups), but the PCE displayed modest evidence of miscalibration (calibration statistic K=43.7, miscalibration from 9% to 31%). Discrimination was similar in both models (C‐index=0.740 for FRS, 0.747 for PCE). Refitting the published models using EHD did not substantially improve calibration or discrimination.ConclusionsWe conclude that published cardiovascular risk models can be successfully applied to EHD to estimate cardiovascular risk; the FRS remains valid and is not obsolete; and model refitting does not meaningfully improve the accuracy of risk estimates.

Computational models and neural nets: Fantastic models—Where to find them and how to identify them

“…The next development in the 1990s was the beginning of the use of mathematical models based on logistic regression analyses of epidemiological datasets once semiconductor‐based scientific notation calculators became available. These could be simplified into paper‐based systems or mechanical tools for routine clinical use . Now that substantial computing capacity is available through cell phones or internet‐based systems these are now universally recommended for assessment of patients with a risk of CVD.…”

“…The main methods used have been to use larger more representative datasets based either on aggregating epidemiological cohort studies (eg US atherosclerotic CVD score‐ ASCVD) or national EHRs (eg QRISK in the UK). The best predictive performance of epidemiological datasets is an average area under curve (AUC) for receiver operator characteristic (ROC) curves (ie C‐statistic) of approximately 0.70‐0.75 . Adding imaging data from coronary artery calcium increases this to 0.79 with less benefit from the far more convenient ultrasound techniques or biomarkers such as high‐sensitivity troponin measurements …”

“…These could be simplified into paper-based systems or mechanical tools for routine clinical use. 10,11 Now that substantial computing capacity is available through cell phones or internet-based systems these are now universally recommended for assessment of patients with a risk of CVD. The desire to increase convenience has now led to the wish to simplify the process further by abolishing the most logistically difficult (and expensive) aspect which comprises the cholesterol blood tests.…”

“…The best predictive performance of epidemiological datasets is an average area under curve (AUC) for receiver operator characteristic (ROC) curves (ie C-statistic) of approximately 0.70-0.75. 11 Adding imaging data from coronary artery calcium increases this to 0.79 with less benefit from the far more convenient ultrasound techniques or biomarkers such as high-sensitivity troponin measurements. 15,16 The next great hope is to exploit the developments in electronic databases and advances in computing.…”

Ending the epidemics of cardiovascular diseases: time is now to integrate both population and individual strategies for prevention

“…Incorporation of these markers beyond those traditionally used in risk prediction significantly improved the discrimination ability and is a novel and practical approach in developing population-specific risk prediction models. Future investigations of similar nature may also consider other predictors such as inflammatory markers, genetic data, or even data from electronic health records [15], in order to further improve personalized prediction performance. However, one must keep in mind that incorporation of additional predictors increases the complexity of the models and difficulty of implementation in general settings.…”