A calibration hierarchy for risk models was defined: from utopia to empirical data

Calster, Ben Van; Nieboer, Daan; Vergouwe, Yvonne; Cock, B. De; Pencina, Michael J.; Steyerberg, Ewout W.

doi:10.1016/j.jclinepi.2015.12.005

Cited by 584 publications

(648 citation statements)

References 32 publications

Supporting

Mentioning

643

Contrasting

Unclassified

Order By: Relevance

“…A calibration slope of less than 1 indicates overfitting of the original prognostic models. 46 A history of gestational diabetes mellitus is an important predictor in most models, but is always scored as negative (=zero) in the prognostic algorithms for nulliparous women, owing to them not having had a previous pregnancy. Therefore, we also reassessed discrimination and calibration of all 12 logistically recalibrated models in a subgroup analysis of nulliparous women to see if the results of the whole population were not merely the result of an excellent prediction of gestational diabetes mellitus in multiparous women (that is, women with a known history of the disorder).…”

Section: Discussionmentioning

confidence: 99%

External validation of prognostic models to predict risk of gestational diabetes mellitus in one Dutch cohort: prospective multicentre cohort study

Ruiter

Kwee

Naaktgeboren

et al. 2016

BMJ

View full text Add to dashboard Cite

ObjeCtiveTo perform an external validation and direct comparison of published prognostic models for early prediction of the risk of gestational diabetes mellitus, including predictors applicable in the first trimester of pregnancy. DesignExternal validation of all published prognostic models in large scale, prospective, multicentre cohort study. setting 31 independent midwifery practices and six hospitals in the Netherlands. PartiCiPantsWomen recruited in their first trimester (<14 weeks) of pregnancy between December 2012 and January 2014, at their initial prenatal visit. Women with pre-existing diabetes mellitus of any type were excluded. Main OutCOMe MeasuresDiscrimination of the prognostic models was assessed by the C statistic, and calibration assessed by calibration plots. results 3723 women were included for analysis, of whom 181 (4.9%) developed gestational diabetes mellitus in pregnancy. 12 prognostic models for the disorder could be validated in the cohort. C statistics ranged from 0.67 to 0.78. Calibration plots showed that eight of the 12 models were well calibrated.The four models with the highest C statistics included almost all of the following predictors: maternal age, maternal body mass index, history of gestational diabetes mellitus, ethnicity, and family history of diabetes. Prognostic models had a similar performance in a subgroup of nulliparous women only. Decision curve analysis showed that the use of these four models always had a positive net benefit. COnClusiOnsIn this external validation study, most of the published prognostic models for gestational diabetes mellitus show acceptable discrimination and calibration. The four models with the highest discriminative abilities in this study cohort, which also perform well in a subgroup of nulliparous women, are easy models to apply in clinical practice and therefore deserve further evaluation regarding their clinical impact. IntroductionIn the field of obstetrics, the number of publications on prognostic models has more than tripled in the past decade, 1 which reflects an increasing interest in risk based medicine. Risk based medicine aims to provide the most appropriate care to each patient, often guided by outcome risk estimates based on individual patient characteristics, test results, or even genetic information. 2 As a result of the obesity pandemic, the incidence of gestational diabetes mellitus, notably occurring in the second or third trimester, is rising and is increasingly contributing to perinatal complications such as macrosomia, shoulder dystocia, caesarean section, and neonatal hypoglycaemia. 3 4 Moreover, long term sequelae of gestational diabetes mellitus are type 2 diabetes in mothers and obesity in their offspring. 5 6 Early diagnosis and treatment of gestational diabetes mellitus have been proven to improve pregnancy outcomes. 7 8 Some guidelines propose a population strategy for diagnosing the disorder [9][10][11][12] (that is, an oral glucose tolerance test) in all pregnant women, whereas others opt for a high ri...

show abstract

Section: Discussionmentioning

confidence: 99%

External validation of prognostic models to predict risk of gestational diabetes mellitus in one Dutch cohort: prospective multicentre cohort study

Ruiter

Kwee

Naaktgeboren

et al. 2016

BMJ

View full text Add to dashboard Cite

show abstract

“…29 Model calibration was evaluated by the Greenwood-Nam-D’Agostino (GND) approach, which analyzes calibration of models on four levels using a hierarchy of increasing strictness (corresponding to mean, weak, moderate, or strong calibration). 30,31 The first level (mean calibration) measures whether the overall average predicted risk equals the observed event rate. The second level (weak calibration) evaluates the slope and intercept of the models; the closer to 1 the slope and closer to 0 the intercept, the better the calibration.…”

Section: Methodsmentioning

confidence: 99%

Assessing and Refining Myocardial Infarction Risk Estimation Among Patients With Human Immunodeficiency Virus

et al. 2017

View full text Add to dashboard Cite

Importance Persons with human immunodeficiency virus (HIV) treated with antiretroviral therapy (ART) have improved longevity but are at elevated risk for myocardial infarction (MI) due to common MI risk factors and HIV-specific factors. Despite these elevated MI rates, optimal methods to predict MI risks for HIV-infected persons remain unclear. Objective To determine the extent to which existing and de novo estimation tools predict MI in a multi-center HIV cohort with rigorous MI adjudication. Design We evaluated the performance of standard-of-care and two new data-derived MI risk estimation models in the Centers for AIDS Research Network of Integrated Clinical Systems (CNICS) multi-center prospective clinical cohort. The new risk estimation models were validated in a cohort separate from the derivation cohort. Setting Clinical sites across the U.S. where HIV-infected adults receive medical care in inpatient and outpatient settings. Participants HIV-infected adults receiving care anytime since 1995 at 5 CNICS sites where MIs were adjudicated (N=19829). Exposures Common cardiovascular risk factors, HIV viral load, CD4 count, and medication use were used to calculate predicted event rates. Main Outcome and Measures Observed MI rates over the course of follow-up, scaled to 10 years using an observed prime approach to account for dropout and loss to follow-up prior to 10 years. Results MI rates were higher among blacks, older participants, and participants who were not virally suppressed. The 2013 Pooled Cohort Equations (PCEs), which predict composite rates of MI and stroke, adequately discriminated MI risk (Harrell’s C Statistic = 0.75). Two data-derived models incorporating HIV-specific covariates exhibited weak calibration in a validation sample and did not discriminate risk any better (Harrell’s C Statistic = 0.72 and 0.73) than the PCEs. The PCEs were moderately calibrated in CNICS but predicted consistently lower than observed prime rates of MI. The PCEs Conclusions and relevance The PCEs discriminated MI risk and were moderately calibrated in this multi-center HIV cohort. Adding HIV-specific factors did not improve model performance. As HIV-infected cohorts capture and assess outcomes of MI and stroke, the performance of risk estimation tools should be revisited.

show abstract

“…The aforementioned intercept and slope narrowly deviate from the ideal values. For this reason, the calibration plots should be investigated, however, according to Van Calster et al, a graphical assessment should be done with at least 200 events and 200 nonevents(Van Calster et al, 2016b). This condition is not fulfilled within the study group; hence, we omit a detailed analysis of the plots (Fig.…”

Section: Calibrationmentioning

confidence: 99%

Response to letter to the editor concerning validation of IOTA ADNEX model

Szubert

Wójtowicz

Żywica

2016

Gynecologic Oncology Reports

View full text Add to dashboard Cite

A calibration hierarchy for risk models was defined: from utopia to empirical data

Abstract: Strong calibration is desirable for individualized decision support but unrealistic and counter productive by stimulating the development of overly complex models. Model development and external validation should focus on moderate calibration.

Cited by 584 publications

References 32 publications

External validation of prognostic models to predict risk of gestational diabetes mellitus in one Dutch cohort: prospective multicentre cohort study

External validation of prognostic models to predict risk of gestational diabetes mellitus in one Dutch cohort: prospective multicentre cohort study

Assessing and Refining Myocardial Infarction Risk Estimation Among Patients With Human Immunodeficiency Virus

Response to letter to the editor concerning validation of IOTA ADNEX model

Contact Info

Product

Resources

About