Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective

Abstract

It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out‐of‐sample performance can be hampered when predictors are measured differently at derivation and external validation. This may occur, for instance, when predictors are measured using different measurement protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement between derivation and validation data is common, the impact on the out‐of‐sample performance is not well studied. Using analytical and simulation approaches, we examined out‐of‐sample performance of prediction models under various scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity can induce miscalibration of prediction and affects discrimination and overall predictive accuracy, to extents that the prediction model may no longer be considered clinically useful. The measurement error taxonomy was found to be helpful in identifying and predicting effects of heterogeneous predictor measurements between settings of prediction model derivation and validation. Our work indicates that homogeneity of measurement strategies across settings is of paramount importance in prediction research.

Predicting Prostate Cancer Death with Different Pretreatment Risk Stratification Tools: A Head-to-head Comparison in a Nationwide Cohort Study

“…Measurement heterogeneity refers to differences in the procedure and/or instruments used to measure the predictors. It has been shown that prognostic models including unreliable/misclassified predictors perform suboptimally on internal (271,272) and, especially, external validation (273)(274)(275). In prostate cancer, all standard variables used for developing prognostic models are known to be unreliably measured and/or are subject to measurement heterogeneity.…”

Changing predictor measurement procedures affected the performance of prediction models in clinical examples

“…Furthermore, we take the position that addressing measurement heterogeneity at the data collection stage is preferred over statistical correction for measurement error in predictors. Correctionsdtypically aiming to alleviate measurement-error bias in regression coefficientsdmay increase rather than reduce the measurement heterogeneity [12].…”

“…We refer to these differences in measurement across settings as predictor measurement heterogeneity. Simulation studies have shown that predictor measurement heterogeneity can induce miscalibration of prediction models and affect discrimination and accuracy at external validation [12]. Although predictor measurement heterogeneity across derivation and validation samples appears to be common in clinical (research) settings (see, e.g., studies by Collins et al [4], Te Velde et al [15], and Smith et al [16]), its impact on the performance of prediction models at validation is not well studied using empirical data.…”

Prediction of contralateral breast cancer: external validation of risk calculators in 20 international cohorts

“…Besides, inter-observer variation in pathological examination of BC among studies may lead to different adjuvant systemic therapy advice and, consequently, prediction of CBC risk [23]. Variation in prediction performance and limited generalizability of CBC risk calculators can also be partially explained by differences in how predictors are measured among studies [24,25]. For example, lack of family history knowledge may lead to uncertainty in risk prediction and varies according to demographics of the patients [26].…”