Calibration strategies to validate predictive models: is new always better?

Abstract

Calibration along with discrimination is an important measure of accuracy to validate predictive logistic regression models. Most predictive models in intensive care such as Simplified Acute Physiology Score (SAPS) II [1] and SAPS 3 [2,3] consider the binary outcome whether a patient will be alive or dead at hospital discharge. Discrimination measures how well the model can distinguish between patients who die and those who survive. Discrimination is usually assessed by the area under the receiver operating characteristic curve (AU-ROC) [4]. This statistic evaluates each pair of observations that have different outcomes and calculates the proportion of times when the patient who died had a higher predicted mortality than did the survivor. The AU-ROC ranges from 0.50 (no discrimination: complete binary random of 50 % similar to flipping a coin) to 1.00 (100 % correct discrimination of the model) [4].Calibration measures the model's ability to generate predictions that are on average close to the average observed outcome. Calibration has traditionally been approached in two steps. First, to investigate the overall ability to correctly relate the actual occurrence of the event to its estimated probability using statistical methods. The most widely used method is the HosmerLemeshow (H-L) test [5], which examines how well the percentage of observed deaths matches the percentage of predicted deaths over deciles of predicted risk. A p value greater than 0.05 is needed to conclude that there are no significant differences between the observed and expected outcomes and therefore the model has good overall calibration. Second, to localize possible deviations across risk strata by means of calibration plots with observed outcomes versus expected probabilities of mortality. The calibration plot, also named calibration curve, intends to provide complementary information over subsets. If the model calibrates well, there will not be a substantial deviation from the 45°line of perfect fit or bisector. On the contrary, miscalibration of the model will be a function of expected probability.The H-L test is easy to compute and its interpretation is intuitive, but it has acknowledged limitations such as being very sensitive to sample size [6][7][8]. The traditional plot or calibration curve also has some disadvantages: first, rather than a curve, it is a jagged line connecting the points in the plot; second, it is not accompanied by any information on the statistical significance of deviations from the bisector [9].Methods to measure the calibration of predictive models have remained unchanged for a long time, but new insights are now provided. In this issue of Intensive Care Medicine, Poole et al.[10] present a multicenter study aimed at comparing the performance of SAPS II and SAPS 3 in predicting hospital mortality. Discrimination was measured as usual using ROC curves and both models discriminated fairly. Interestingly, the calibration ability of the systems was measured using a new approach named calibration belt [9].Both syste...

Predicting six-month mortality of patients with traumatic brain injury: usefulness of common intensive care severity scores

“…The GiViTI belt has two main applications: performance comparison between different centers and external validation of prediction models [30,35,44]. Although the mathematical basis of the GiViTI calibration belt has been shown elsewhere it should be compared to the H-L test, which has been considered the gold standard of calibration testing [31,35]. We found the GiViTI and H-L tests to generate similar results for calibration.…”

“…This is due to the higher degree of polynomial function fitted between the predicted and observed outcome, resulting in wide confidence intervals [35]. This is to our knowledge one of the first clinical studies comparing the traditional H-L calibration test with the GiViTI calibration belt [31]. The GiViTI calibration belt should be strongly considered in further studies in addition to the traditional H-L test.…”

“…To overcome the limitations of the H-L, we combined the classic H-L test with a new statistical test for calibration, the GiViTI calibration belt [30,31]. In addition to giving a calibration curve that illustrates the relationship between predicted risk and observed outcome over different levels of risk, this technique also gives the confidence belt of the curve, that is, an estimation of the degree of uncertainty regarding the true location of the curve.…”

Prognostic models in traumatic brain injury

“…Studies II and IV are, however, among the fi rst to assess and compare the H-L and GiViTI calibration tests. 240 Th e main advantage of the GiViTI calibration belt over the H-L test and the calibration slope is the possibility of estimating 95% confi dence intervals over the whole risk spectrum. Th us, the calibration belt provides valuable information about the degree and direction of miscalibration, such as whether it aff ects only a specifi c risk interval or if the overall calibration is poor.…”

On Assessing Binary Regression Models Based on Ungrouped Data

“…The HL test has been routinely used in logistic regression analysis and is commonly included/taught in textbooks/courses on logistic regression (e.g., Agresti, ; Hosmer et al, ). In the particular application of calibration of mortality benchmarks in critical care, for instance, the HL test is the most widely used method (Serrano, ). Issues of possible low power of the HL test and its sensitivity to data grouping have been raised (Hosmer et al, ).…”