How to evaluate the calibration of a disease risk prediction tool

Abstract

To evaluate the calibration of a disease risk prediction tool, the quantity E/O, i.e. the ratio of the expected to the observed number of events, is usually computed. However, because of censoring, or more precisely because of individuals who drop out before the termination of the study, this quantity is generally unavailable for the complete population study and an alternative estimate has to be computed. In this paper, we present and compare four methods to do this. We show that two of the most commonly used methods generally lead to biased estimates. Our arguments are first based on some theoretic considerations. Then, we perform a simulation study to highlight the magnitude of biases. As a concluding example, we evaluate the calibration of an existing predictive model for breast cancer on the E3N-EPIC cohort.

Estimating the performance of three cardiovascular disease risk scores: the Estonian Biobank cohort study

“…Lastly, we would like to stress the importance of proper methodology for adjusting the risk scores to correspond to a follow-up time that is less than 10 years. In several publications, the follow-up times have been modified independently of the actual event status, leading to biases in the SIR estimation 28. Here, the aforementioned bias was accounted for (details in the online supplement).…”

Soluble Urokinase Plasminogen Activator Receptor for Risk Prediction in Patients Admitted with Acute Chest Pain

“…The expected numbers of events and nonevents were used in the estimation of NRI to account for censored data and calculated by multiplying the total number of people by the Kaplan-Meier rates at the end of follow-up. This approach was found optimal for assessing calibration of survival models (21 ). We estimated the bias-corrected CIs for NRI by bootstrap resampling (1000 replicates).…”

Discrimination measures for survival outcomes: Connection between the AUC and the predictiveness curve

“…To our knowledge, the Hosmer-Lemeshow statistic has never been extended to survival outcomes. However, as mentioned in Viallon et al (2009), this statistic can be computed with observed counts replaced by their estimates based on Kaplan-Meier estimator. Some more study would be needed to assess the asymptotical distribution of the resulting statistic under the null hypothesis in order to derive a proper statistical test.…”

“…Most of the existing criteria were originally defined for diagnostic tests and rely on the observation of a binary outcome representing, for instance, disease status. Extending these criteria to survival outcomes is generally not straightforward, especially because of the presence of censored data (see, e.g., Viallon et al (2009) for the calibration of risk prediction tools). In the present work we will focus on methods that assess discrimination of prognostic tools.…”