Validation of discrete time‐to‐event prediction models in the presence of competing risks

Abstract

Clinical prediction models play a key role in risk stratification, therapy assignment and many other fields of medical decision making. Before they can enter clinical practice, their usefulness has to be demonstrated using systematic validation. Methods to assess their predictive performance have been proposed for continuous, binary, and time-to-event outcomes, but the literature on validation methods for discrete time-toevent models with competing risks is sparse. The present paper tries to fill this gap and proposes new methodology to quantify discrimination, calibration, and prediction error (PE) for discrete time-to-event outcomes in the presence of competing risks. In our case study, the goal was to predict the risk of ventilator-associated pneumonia (VAP) attributed to Pseudomonas aeruginosa in intensive care units (ICUs). Competing events are extubation, death, and VAP due to other bacteria. The aim of this application is to validate complex prediction models developed in previous work on more recently available validation data. K E Y W O R D Sarea under the curve, calibration slope, competing events, discrete time-to-event model, dynamic prediction models, prediction error, validation INTRODUCTIONClinical prediction models aim to give valid outcome predictions for new patients and to provide a good basis for treatment decisions. Such models need to be systematically validated before entering clinical practice. Assessing the predictive performance in the data set from which the model has been derived will most certainly give an assessment which is too optimistic. To avoid this issue, some kind of cross-validation or external validation is needed. In perfect conditions, the performance of the prediction model is assessed in a second independent data set (Steyerberg, 2009). Such validation data, also referred to as testing data, should incorporate new patients from a different time period or patients from a different center. To quantify how well the prediction model performs, commonly used measures of discrimination and calibration can be computed. A model has satisfactory discrimination if it is able to adequately discriminate between cases and controls. Moreover, a well-calibrated model guarantees good agreement between observed outcomes and predictions. Finally, to evaluate overall performance, quadratic scoring rules like the prediction error (PE) or Brier score (BS) can be calculated to simultaneously assess calibration and discrimination.This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

Dementia risk in the general population: large-scale external validation of prediction models in the AGES-Reykjavik study

“…2 Distribution of c statistics across studies with 95% confidence intervals (*model was developed for more than 1 outcome or time horizon) Fig. 3 Calibration plots of (re)calibrated prognostic models to predict risk of (a) all-cause dementia and (b) Alzheimer's disease; an intercept of 0 and slope of 1 (i.e., the diagonal line) represents ideal cali-bration and more spread between the groups indicates better model performance than less spread-error bars in grouped observations represent 95% confidence intervals; Q = quartile When externally evaluating the performance of a prediction model, both discrimination and calibration should be assessed: good performance on one measure does not ensure good performance on the other [34]. Among the models that we externally validated, best performance combining both discrimination and calibration was achieved by the original model by Hogan et al [24] and the recalibrated 2009 model by Barnes et al [25] In our validation data, proxies had to be used for several predictors in both models; despite these proxies, the models obtained acceptable discrimination and good (re)calibration.…”

Development and validation of a nomogram for predicting severity in patients with hemorrhagic fever with renal syndrome: A retrospective study

“…We evaluated the performance of the nomogram through discrimination and calibration in the training population and the veri cation population, respectively. Since the consistency index (C-index) is equivalent to the area under the receiver operating characteristic curve (AUC) in logistic regression, we use the AUC to evaluate the discriminative ability of the nomogram [13] . The Hosmer-Lemeshow goodness-oft test is used to evaluate the calibration of the nomogram, and a calibration curve is drawn to visualize the consistency between the predicted results and the observed results [14] .…”

A Predicting Nomogram for Mortality in Patients With COVID-19

“…We tested the accuracy of the nomograms by discrimination and calibration both in the primary and the validation cohort. We used the receiver operating characteristic curve (ROC) to assess the discriminative ability of the nomogram and then assessed the area under the curve (AUC) (14,15). Calibration curves were used to compare the association between actual outcomes and predicted probabilities (16).…”