Measurement error and timing of predictor values for multivariable risk prediction models are poorly reported

Whittle, Rebecca; Peat, George; Belcher, John; Collins, Gary S.; Riley, Richard D

doi:10.1016/j.jclinepi.2018.05.008

Cited by 21 publications

(13 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some researchers recommend that the reliability of predictor variables be explicitly evaluated, and that only those demonstrating good agreement beyond that expected by chance alone should be considered for inclusion [17]. A recent study found that measurement error of predictor variables is poorly reported, and that researchers seldom state explicitly when the predictors should be measured, and the CPR applied [140]. Another study demonstrated that predictor measurement heterogeneity across settings can have a detrimental impact on the performance of a CPR at external validation [141].…”

Section: Stages In the Development Of Clinical Prediction Rulesmentioning

confidence: 99%

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

et al. 2019

View full text Add to dashboard Cite

Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.

show abstract

Section: Stages In the Development Of Clinical Prediction Rulesmentioning

confidence: 99%

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

et al. 2019

View full text Add to dashboard Cite

show abstract

“…By taking the measurement error perspective, our study revealed that prediction research requires consideration of variation in measurement procedures across different settings of derivation and validation, rather than analyzing the amount of measurement error within a study. A recent systematic review by Whittle and colleagues demonstrated that measurement error was not acknowledged in many prediction studies, and pointed out the need to investigate consequences of measurement error in prediction research . An important starting point for this research following from our study is that the generalizability of prediction models depends on the transportability of measurement structures.…”

Section: Discussionmentioning

confidence: 79%

Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective

Luijken

Groenwold

Calster

et al. 2019

Statistics in Medicine

View full text Add to dashboard Cite

It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out‐of‐sample performance can be hampered when predictors are measured differently at derivation and external validation. This may occur, for instance, when predictors are measured using different measurement protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement between derivation and validation data is common, the impact on the out‐of‐sample performance is not well studied. Using analytical and simulation approaches, we examined out‐of‐sample performance of prediction models under various scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity can induce miscalibration of prediction and affects discrimination and overall predictive accuracy, to extents that the prediction model may no longer be considered clinically useful. The measurement error taxonomy was found to be helpful in identifying and predicting effects of heterogeneous predictor measurements between settings of prediction model derivation and validation. Our work indicates that homogeneity of measurement strategies across settings is of paramount importance in prediction research.

show abstract

“…Measurement heterogeneity within development and validation studies, for example, because of variability in measurement precision between clinicians or centers [37], is an important topic for future research. Given the potential impact and limited attention to date [38], research is needed on the effect of measurement heterogeneity for other statistical models and outcomes (e.g., survival models for time-to-event outcomes) and the impact on more flexible prediction modeling strategies. Finally, the similarity between the preferred and pragmatic measurement of a predictor was quantified using a partial correlation coefficient.…”

Section: Discussionmentioning

confidence: 99%

Changing predictor measurement procedures affected the performance of prediction models in clinical examples

Luijken

Wynants

Smeden

et al. 2020

Journal of Clinical Epidemiology

View full text Add to dashboard Cite

Objectives: The aim of this study was to quantify the impact of predictor measurement heterogeneity on prediction model performance. Predictor measurement heterogeneity refers to variation in the measurement of predictor(s) between the derivation of a prediction model and its validation or application. It arises, for instance, when predictors are measured using different measurement instruments or protocols. Study Design and Setting: We examined the effects of various scenarios of predictor measurement heterogeneity in real-world clinical examples using previously developed prediction models for diagnosis of ovarian cancer, mutation carriers for Lynch syndrome, and intrauterine pregnancy. Results: Changing the measurement procedure of a predictor influenced the performance at validation of the prediction models in nine clinical examples. Notably, it induced model miscalibration. The calibration intercept at validation ranged from À0.70 to 1.43 (0 for good calibration), whereas the calibration slope ranged from 0.50 to 1.67 (1 for good calibration). The difference in Cstatistic and scaled Brier score between derivation and validation ranged from À0.08 to þ0.08 and from À0.40 to þ0.16, respectively. Conclusion: This study illustrates that predictor measurement heterogeneity can influence the performance of a prediction model substantially, underlining that predictor measurements used in research settings should resemble clinical practice. Specification of measurement heterogeneity can help researchers explaining discrepancies in predictive performance between derivation and validation setting.

show abstract

Measurement error and timing of predictor values for multivariable risk prediction models are poorly reported

Abstract: Reporting of measurement error and intended moment of model use is poor in prediction model studies. There is a need to identify circumstances where ignoring measurement error in prediction models is consequential and whether accounting for the error will improve the predictions.

Cited by 21 publications

References 79 publications

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective

Changing predictor measurement procedures affected the performance of prediction models in clinical examples

Contact Info

Product

Resources

About