Do population-level risk prediction models that use routinely collected health data reliably predict individual risks?

Li, Yan; Sperrin, Matthew; Belmonte, Miguel; Pate, Alexander; Ashcroft, Darren M.; Staa, Tjeerd van

doi:10.1038/s41598-019-47712-5

Cited by 24 publications

(38 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We used the Markov chain Monte Carlo method with monotone style to impute missing values 10 times for ethnicity ( 18 (only these variables had missing values). We randomly split the overall cohort (which contained 10 imputations) into an overall derivation cohort (75%) and an overall testing cohort (25%).…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar

Sperrin

Ashcroft

et al. 2020

BMJ

Self Cite

View full text Add to dashboard Cite

ObjectiveTo assess the consistency of machine learning and statistical techniques in predicting individual level and population level risks of cardiovascular disease and the effects of censoring on risk predictions.DesignLongitudinal cohort study from 1 January 1998 to 31 December 2018.Setting and participants3.6 million patients from the Clinical Practice Research Datalink registered at 391 general practices in England with linked hospital admission and mortality records.Main outcome measuresModel performance including discrimination, calibration, and consistency of individual risk prediction for the same patients among models with comparable model performance. 19 different prediction techniques were applied, including 12 families of machine learning models (grid searched for best models), three Cox proportional hazards models (local fitted, QRISK3, and Framingham), three parametric survival models, and one logistic model.ResultsThe various models had similar population level performance (C statistics of about 0.87 and similar calibration). However, the predictions for individual risks of cardiovascular disease varied widely between and within different types of machine learning and statistical models, especially in patients with higher risks. A patient with a risk of 9.5-10.5% predicted by QRISK3 had a risk of 2.9-9.2% in a random forest and 2.4-7.2% in a neural network. The differences in predicted risks between QRISK3 and a neural network ranged between –23.2% and 0.1% (95% range). Models that ignored censoring (that is, assumed censored patients to be event free) substantially underestimated risk of cardiovascular disease. Of the 223 815 patients with a cardiovascular disease risk above 7.5% with QRISK3, 57.8% would be reclassified below 7.5% when using another model.ConclusionsA variety of models predicted risks for the same patients very differently despite similar model performances. The logistic models and commonly used machine learning models should not be directly applied to the prediction of long term risks without considering censoring. Survival models that consider censoring and that are explainable, such as QRISK3, are preferable. The level of consistency within and between models should be routinely assessed before they are used for clinical decision making.

show abstract

Section: Discussionmentioning

confidence: 99%

“…24 This was different from QRISK3, for which a single calendar time date was mostly used. 18 The main inclusion criteria were age between 25 and 84 years, no history of cardiovascular disease, and no prescription for a statin before the index date. The outcome of interest was the 10 year risk of developing cardiovascular disease.…”

Section: Data Sourcementioning

confidence: 99%

Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar

Sperrin

Ashcroft

et al. 2020

BMJ

Self Cite

View full text Add to dashboard Cite

show abstract

“…biomarkers) or practice level, which could reduce the unmeasured heterogeneity in CVD incidence across practices. Further research could consider more individual level based methods, such as a Bayesian clinical reasoning model 33 and machine learning models 34 , as this study and other findings 18,35,36 show that Cox models with similar conventional model performance metrics (C-stat 37…”

Section: Implications For Research and Practicementioning

confidence: 54%

Examining the impact of data quality and completeness of electronic health records on predictions of patients’ risks of cardiovascular disease

Li¹,

Sperrin²,

Martin³

et al. 2020

International Journal of Medical Informatics

Self Cite

View full text Add to dashboard Cite

ObjectiveTo assess the extent of variation of data quality and completeness of electronic health records and impact on the robustness of risk predictions of incident cardiovascular disease (CVD) using a risk prediction tool that is based on routinely collected data (QRISK3). Design:Longitudinal cohort study.Setting 392 general practices (including 3.6 million patients) linked to hospital admission data. MethodsVariation in data quality was assessed using Sáez's stability metrics quantifying outlyingness of each practice. Statistical frailty models evaluated whether accuracy of QRISK3 predictions on individual predictions and effects of overall risk factors (linear predictor) varied between practices. ResultsThere was substantial heterogeneity between practices in CVD incidence unaccounted for by QRISK3. In the lowest quintile of statistical frailty, a QRISK3 predicted risk of 10% for female was in a range between 7.1% and 9.0% when incorporating practice variability into the statistical frailty models; for the highest quintile, this was 10.9%-16.4%. Data quality (using Saez metrics) and completeness were comparable across different levels of statistical frailty. For example, recording of missing information on ethnicity was 55.7%, 62.7%, 57.8%, 64.8% and 62.1% for practices from lowest to 3 highest quintiles of statistical frailty respectively. The effects of risk factors did not vary between practices with little statistical variation of beta coefficients. ConclusionsThe considerable unmeasured heterogeneity in CVD incidence between practices was not explained by variations in data quality or effects of risk factors. QRISK3 risk prediction should be supplemented with clinical judgement and evidence of additional risk factors.

show abstract

“…QRISK3 is the most popular risk prediction model for CVD developed in the UK. It calculates risk of patients developing CVD in the next 10 years and has been incorporated into the electronic health records (EHRs) system in the UK in order to detect high risk CVD patients and help clinicians make treatment decisions 3,4 . NICE guidelines recommend clinicians to consider prescribing statins to patients with a risk over 10% identified from QRISK3 5 .…”

Section: Introductionmentioning

confidence: 99%

“…NICE guidelines recommend clinicians to consider prescribing statins to patients with a risk over 10% identified from QRISK3 5 . QRISK3 was developed from historical patients' EHR data using Cox proportional hazard model 6 and has been well validated at population level corresponding to discrimination and calibration 3,4,7 .…”

Section: Introductionmentioning

confidence: 99%

R package “QRISK3”: an unofficial research purposed implementation of ClinRisk’s QRISK3 algorithm into R

Sperrin

Staa

2020

F1000Res

Self Cite

View full text Add to dashboard Cite

Cardiovascular disease has been the leading cause of death for decades. Risk prediction models are used to identify high risk patients; the most common model used in the UK is ClinRisk’s QRISK3. In this paper we describe the implementation of the QRISK3 algorithm into an R package. The package was successfully validated by the open sourced QRISK3 algorithm and QRISK3 SAS program. We provide detailed examples of the use of the package, including assigning QRISK3 scores for a large cohort of patients. This R package could help the research community to better understand risk prediction scores and improve future risk prediction models. The package is available from CRAN: https://cran.r-project.org/web/packages/QRISK3/index.html.

show abstract

Do population-level risk prediction models that use routinely collected health data reliably predict individual risks?

Cited by 24 publications

References 27 publications

Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar

Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar

Examining the impact of data quality and completeness of electronic health records on predictions of patients’ risks of cardiovascular disease

R package “QRISK3”: an unofficial research purposed implementation of ClinRisk’s QRISK3 algorithm into R

Contact Info

Product

Resources

About