Changing predictor measurement procedures affected the performance of prediction models in clinical examples

Abstract

Objectives: The aim of this study was to quantify the impact of predictor measurement heterogeneity on prediction model performance. Predictor measurement heterogeneity refers to variation in the measurement of predictor(s) between the derivation of a prediction model and its validation or application. It arises, for instance, when predictors are measured using different measurement instruments or protocols. Study Design and Setting: We examined the effects of various scenarios of predictor measurement heterogeneity in real-world clinical examples using previously developed prediction models for diagnosis of ovarian cancer, mutation carriers for Lynch syndrome, and intrauterine pregnancy. Results: Changing the measurement procedure of a predictor influenced the performance at validation of the prediction models in nine clinical examples. Notably, it induced model miscalibration. The calibration intercept at validation ranged from À0.70 to 1.43 (0 for good calibration), whereas the calibration slope ranged from 0.50 to 1.67 (1 for good calibration). The difference in Cstatistic and scaled Brier score between derivation and validation ranged from À0.08 to þ0.08 and from À0.40 to þ0.16, respectively. Conclusion: This study illustrates that predictor measurement heterogeneity can influence the performance of a prediction model substantially, underlining that predictor measurements used in research settings should resemble clinical practice. Specification of measurement heterogeneity can help researchers explaining discrepancies in predictive performance between derivation and validation setting.

Predicting Prostate Cancer Death with Different Pretreatment Risk Stratification Tools: A Head-to-head Comparison in a Nationwide Cohort Study

“…Measurement heterogeneity refers to differences in the procedure and/or instruments used to measure the predictors. It has been shown that prognostic models including unreliable/misclassified predictors perform suboptimally on internal (271,272) and, especially, external validation (273)(274)(275). In prostate cancer, all standard variables used for developing prognostic models are known to be unreliably measured and/or are subject to measurement heterogeneity.…”

An Electronic Health Record–Compatible Model to Predict Personalized Treatment Effects From the Diabetes Prevention Program: A Cross-Evidence Synthesis Approach Using Clinical Trial and Real-World Data

“…The bottom graphs show effects on the absolute risk difference scale, which shows increasing benefits for higher-risk patients for both interventions. 23,38 Finally, use of trial or registry data cannot yield a model robust to missing values in the EHR database used for clinical prediction because the pattern of missingness present across research and EHR environments is expected to differ. The usual approaches addressing potential bias arising from missingness (eg, multiple imputation) are not designed to cope with missingness in variables used to generate predictions.…”

An Electronic Health Record Compatible Model to Predict Personalized Treatment Effects from the Diabetes Prevention Program: A Cross-Evidence Synthesis Approach Using Clinical Trial and Real World Data

“…Recent work has highlighted that heterogeneity in predictor measurement across different settings can substantially degrade model performance. 21,36 Finally, use of trial or registry data cannot yield a model robust to missing values in the EHR database used for clinical prediction, since the pattern of missingness present across research and EHR environments is expected to differ. The usual approaches addressing potential bias arising from missingness (e.g., multiple imputation) are not designed to cope with missingness in variables used to generate predictions.…”