Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited*

Abstract

Caution should be used in interpreting the calibration of predictive models developed using a smaller data set when applied to larger numbers of patients. A significant Hosmer-Lemeshow test does not necessarily mean that a predictive model is not useful or suspect. While decisions concerning a mortality model's suitability should include the Hosmer-Lemeshow test, additional information needs to be taken into consideration. This includes the overall number of patients, the observed and predicted probabilities within each decile, and adjunct measures of model calibration.

A Scoring System to Predict Unplanned Intubation in Patients Having Undergone Major Surgical Procedures

“…With regards to calibration of the model, the Hosmer-Lemeshow goodness of fit test was significant (p < 0.0001); however, this test has been shown to perform poorly with large sample sizes. [11][12][13] We measured the overall model performance, based on the overall model discrimination and calibration, using the Brier score, where a score of 0 indicates a perfect model and a score of 0.25 indicates a noninformative model. The Brier score was 0.021, indicating good model performance.…”

Calibration strategies to validate predictive models: is new always better?

“…If the model calibrates well, there will not be a substantial deviation from the 45°line of perfect fit or bisector. On the contrary, miscalibration of the model will be a function of expected probability.The H-L test is easy to compute and its interpretation is intuitive, but it has acknowledged limitations such as being very sensitive to sample size [6][7][8]. The traditional plot or calibration curve also has some disadvantages: first, rather than a curve, it is a jagged line connecting the points in the plot; second, it is not accompanied by any information on the statistical significance of deviations from the bisector [9].…”

“…We can argue that the H-L statistics associated with the traditional calibration plot would have provided more or less the same information. In fact, such calibration curve plots are based on strata from H-L goodness-of-fit test and these deciles of risk would have also been able to indicate the direction, extent, and risk classes affected by deviations [5][6][7]. However, we must consider the new approach as a step forward because H-L statistics and traditional calibration plot the average risk in each riskclass, whereas the new method is based on a continuous function that does not average the risk in subgroups.…”

“…Other standard methods and also the H-L test were just used because the theoretical basis they were based on clearly indicated their value and their limits [5,7] and no gold standard exists to be compared with. However, the H-L test was compared to similar ones via simulation under both null and alternative scenarios after their development [7] and also tested again more recently in the same way [6]. Such stress tests have given us new insights into the limitations of the H-L test [6,7].…”

“…However, the H-L test was compared to similar ones via simulation under both null and alternative scenarios after their development [7] and also tested again more recently in the same way [6]. Such stress tests have given us new insights into the limitations of the H-L test [6,7].…”