Caution when using prognostic models: A prospective comparison of 3 recent prognostic models

Nassar, Antônio Paulo; Mocelin, Amílcar Oshiro; Nunes, Andre Luiz Baptiston; Giannini, Fabio; Brauer, Leonardo; Andrade, Fabio Moreira; Dias, Carolina

doi:10.1016/j.jcrc.2011.08.016

Cited by 68 publications

(79 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, ICU profiles vary worldwide, depending on the proportions of medical and surgical patients, admission and discharge policies, availability of intermediate care units and staffing with intensive care specialists. 13 Any transition from a well-established approach to a new one requires caution and validation. Changing APACHE II for SAPS III has some advantages and the most important is the fact that SAPS III is the only prognostic score that included a cohort of patients from South America in its development.…”

Section: -13mentioning

confidence: 99%

Feasibility of transitioning from APACHE II to SAPS III as prognostic model in a Brazilian general intensive care unit. A retrospective study

et al. 2014

View full text Add to dashboard Cite

CONTEXT AND OBJECTIVE: Prognostic models reflect the population characteristics of the countries from which they originate. Predictive models should be customized to fit the general population where they will be used. The aim here was to perform external validation on two predictive models and compare their performance in a mixed population of critically ill patients in Brazil. DESIGN AND SETTING: Retrospective study in a Brazilian general intensive care unit (ICU). METHODS:This was a retrospective review of all patients admitted to a 41-bed mixed ICU from August 2011 to September 2012. Calibration (assessed using the Hosmer-Lemeshow goodness-of-fit test) and discrimination (assessed using area under the curve) of APACHE II and SAPS III were compared. The standardized mortality ratio (SMR) was calculated by dividing the number of observed deaths by the number of expected deaths. RESULTS: A total of 3,333 ICU patients were enrolled. The Hosmer-Lemeshow goodness-of-fit test showed good calibration for all models in relation to hospital mortality. For in-hospital mortality there was a worse fit for APACHE II in clinical patients. Discrimination was better for SAPS III for in-ICU and in-hospital mortality (P = 0.042

show abstract

Section: -13mentioning

confidence: 99%

Feasibility of transitioning from APACHE II to SAPS III as prognostic model in a Brazilian general intensive care unit. A retrospective study

et al. 2014

View full text Add to dashboard Cite

show abstract

“…SAPS 3 overpredicted mortality homogeneously across all risk classes and showed a higher overprediction rate than SAPS II. Similarly, a common finding in most external validation studies is that models show poor calibration, while discrimination is usually good for all of them [11][12][13]. Miscalibration of SAPS 3 has been already reported in several previous studies [11][12][13][14][15][16][17].…”

mentioning

confidence: 67%

“…Similarly, a common finding in most external validation studies is that models show poor calibration, while discrimination is usually good for all of them [11][12][13]. Miscalibration of SAPS 3 has been already reported in several previous studies [11][12][13][14][15][16][17]. Although good performance of SAPS 3 has been shown in some general settings [18,19], in most instances SAPS 3 presents better calibration in specific subsets of critically ill patients [16,18,[20][21][22][23][24][25][26].…”

mentioning

confidence: 77%

Calibration strategies to validate predictive models: is new always better?

Serrano

2012

Intensive Care Med

View full text Add to dashboard Cite

Calibration along with discrimination is an important measure of accuracy to validate predictive logistic regression models. Most predictive models in intensive care such as Simplified Acute Physiology Score (SAPS) II [1] and SAPS 3 [2,3] consider the binary outcome whether a patient will be alive or dead at hospital discharge. Discrimination measures how well the model can distinguish between patients who die and those who survive. Discrimination is usually assessed by the area under the receiver operating characteristic curve (AU-ROC) [4]. This statistic evaluates each pair of observations that have different outcomes and calculates the proportion of times when the patient who died had a higher predicted mortality than did the survivor. The AU-ROC ranges from 0.50 (no discrimination: complete binary random of 50 % similar to flipping a coin) to 1.00 (100 % correct discrimination of the model) [4].Calibration measures the model's ability to generate predictions that are on average close to the average observed outcome. Calibration has traditionally been approached in two steps. First, to investigate the overall ability to correctly relate the actual occurrence of the event to its estimated probability using statistical methods. The most widely used method is the HosmerLemeshow (H-L) test [5], which examines how well the percentage of observed deaths matches the percentage of predicted deaths over deciles of predicted risk. A p value greater than 0.05 is needed to conclude that there are no significant differences between the observed and expected outcomes and therefore the model has good overall calibration. Second, to localize possible deviations across risk strata by means of calibration plots with observed outcomes versus expected probabilities of mortality. The calibration plot, also named calibration curve, intends to provide complementary information over subsets. If the model calibrates well, there will not be a substantial deviation from the 45°line of perfect fit or bisector. On the contrary, miscalibration of the model will be a function of expected probability.The H-L test is easy to compute and its interpretation is intuitive, but it has acknowledged limitations such as being very sensitive to sample size [6][7][8]. The traditional plot or calibration curve also has some disadvantages: first, rather than a curve, it is a jagged line connecting the points in the plot; second, it is not accompanied by any information on the statistical significance of deviations from the bisector [9].Methods to measure the calibration of predictive models have remained unchanged for a long time, but new insights are now provided. In this issue of Intensive Care Medicine, Poole et al.[10] present a multicenter study aimed at comparing the performance of SAPS II and SAPS 3 in predicting hospital mortality. Discrimination was measured as usual using ROC curves and both models discriminated fairly. Interestingly, the calibration ability of the systems was measured using a new approach named calibration belt [9].Both syste...

show abstract

“…They concluded that both scores provided unreliable predictions, but unexpectedly the newer SAPS 3 turned out to overpredict mortality more than the older SAPS II. Consistently, Nassar et al [8] assessed the performance of the APACHE IV, the SAPS 3 and the Mortality Probability Model III [MPM(0)-III] in a population admitted at 3 medical-surgical Brazilian intensive care units and found that all models showed poor calibration, while discrimination was very good for all of them.…”

Section: Introductionmentioning

confidence: 99%

“…To this day, the SAPS II [4] and APACHE II [2] scores remain the most widely used in clinical practice. However, since first being published, they have been modified several times in order to improve their predictive performance [6][7][8][9][10][11]. Despite these extensions of SAPS, predicted hospital mortality remains generally overestimated [8,9,[12][13][14].…”

Section: Introductionmentioning

confidence: 99%

Mortality Prediction in the ICU Based on MIMIC-II Results from the Super ICU Learner Algorithm (SICULA) Project

Pirracchio

2016

Secondary Analysis of Electronic Health Records

View full text Add to dashboard Cite

Learning ObjectivesIn this chapter, we illustrate the use of MIMIC II clinical data, non-parametric prediction algorithm, ensemble machine learning, and the Super Learner algorithm. IntroductionPredicting mortality in patients hospitalized in intensive care units (ICU) is crucial for assessing severity of illness and adjudicating the value of novel treatments, interventions and health care policies. Several severity scores have been developed with the objective of predicting hospital mortality from baseline patient characteristics, defined as measurements obtained within the first 24 h after ICU admission. The first scores proposed, APACHE [1] (Acute Physiology and Chronic Health Evaluation), APACHE II [2], and SAPS [3] (Simplified Acute Physiology Score), relied upon subjective methods for variable importance measure, namely by prompting a panel of experts to select and assign weights to variables according to perceived relevance for mortality prediction. Further scores, such as the SAPS II [4] were subsequently developed using statistical modeling techniques [4][5][6][7]. To this day, the SAPS II [4] and APACHE II [2] scores remain the most widely used in clinical practice. However, since first being published, they have been modified several times in order to improve their predictive performance [6][7][8][9][10][11]. Despite these extensions of SAPS, predicted hospital mortality remains generally overestimated [8,9,[12][13][14]. As an illustration, Poole et al. [9] compared the SAPS II and the SAPS3 performance in a cohort of more than 28,000 admissions to 10 different Italian ICUs. They concluded that both scores provided unreliable predictions, but unexpectedly the newer SAPS 3 turned out to overpredict mortality more than the

show abstract

Caution when using prognostic models: A prospective comparison of 3 recent prognostic models

Cited by 68 publications

References 30 publications

Feasibility of transitioning from APACHE II to SAPS III as prognostic model in a Brazilian general intensive care unit. A retrospective study

Feasibility of transitioning from APACHE II to SAPS III as prognostic model in a Brazilian general intensive care unit. A retrospective study

Calibration strategies to validate predictive models: is new always better?

Mortality Prediction in the ICU Based on MIMIC-II Results from the Super ICU Learner Algorithm (SICULA) Project

Contact Info

Product

Resources

About