Calibrating predictive model estimates to support personalized medicine

Jiang, Xiaoqian; Osl, Melanie; Kim, Jihoon; Ohno-Machado, Lucila

doi:10.1136/amiajnl-2011-000291

Cited by 150 publications

(125 citation statements)

References 28 publications

Supporting

Mentioning

123

Contrasting

Unclassified

Order By: Relevance

“…The target approaches include hidden Markov model (HMM) [38] and one-nearest neighbor classifier with semi-supervised time series classification [39]. We will also consider other metrics, such as classification-based calibration [40, 41], to evaluate and ensure the usefulness of the model. Secondly, other than lab testing results, we would incorporate variables from other categories in the CHOA ICU database, such as more comprehensive ICU stay information, microbiology testing, and procedures.…”

Section: Discussionmentioning

confidence: 99%

icuARM-II

Cheng

Chanani

Maher

et al. 2014

Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics

View full text Add to dashboard Cite

Clinicians in intensive care units (ICUs) rely on standardized scores as risk prediction models to predict a patient’s vulnerability to life-threatening events. Conventional Current scales calculate scores from a fixed set of conditions collected within a specific time window. However, modern monitoring technologies generate complex, temporal, and multimodal patient data that conventional prediction models scales cannot fully utilize. Thus, a more sophisticated model is needed to tailor individual characteristics and incorporate multiple temporal modalities for a personalized risk prediction. Furthermore, most scales models focus on adult patients. To address this needdeficiency, we propose a newly designed ICU risk prediction system, called icuARM-II, using a large-scaled pediatric ICU database from Children’s Healthcare of Atlanta. This novel database contains clinical data collected in 5,739 ICU visits from 4,975 patients. We propose a temporal association rule mining framework giving clinicians a potential to perform predict risks prediction based on all available patient conditions without being restricted by a fixed observation window. We also develop a new metric that can rigidly assesses the reliability of all all generated association rules. In addition, the icuARM-II features an interactive user interface. Using the icuARM-II, our results demonstrated showed a use case of short-term mortality prediction using lab testing results, which demonstrated a potential new solution for reliable ICU risk prediction using personalized clinical data in a previously neglected population.

show abstract

Section: Discussionmentioning

confidence: 99%

icuARM-II

Cheng

Chanani

Maher

et al. 2014

Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics

View full text Add to dashboard Cite

show abstract

“…The practice of using local information to improve diagnostic decision support is not a new idea, 21,22 but has yet to be adopted widely. We have described simplified nearneighbor analysis as one method of multifactorial analysis that is intuitive and relatively easy to interpret.…”

Section: Discussionmentioning

confidence: 99%

Using Patients Like My Patient for Clinical Decision Support: Institution-Specific Probability of Celiac Disease Diagnosis Using Simplified Near-Neighbor Classification

2013

View full text Add to dashboard Cite

“…Recent work by Jiang et al, for example, considered methods that calibrate and in some cases extend models to achieve better calibration. [23]…”

Section: Background and Significancementioning

confidence: 99%

Beyond discrimination: A comparison of calibration methods and clinical usefulness of predictive models of readmission risk

Walsh

Sharman

Hripcsak

2017

Journal of Biomedical Informatics

View full text Add to dashboard Cite

Background Prior to implementing predictive models in novel settings, analyses of calibration and clinical usefulness remain as important as discrimination, but they are not frequently discussed. Calibration is a model’s reflection of actual outcome prevalence in its predictions. Clinical usefulness refers to the utilities, costs, and harms of using a predictive model in practice. A decision analytic approach to calibrating and selecting an optimal intervention threshold may help maximize the impact of readmission risk and other preventive interventions. Objectives To select a pragmatic means of calibrating predictive models that requires a minimum amount of validation data and that performs well in practice. To evaluate the impact of miscalibration on utility and cost via clinical usefulness analyses. Materials and Methods Observational, retrospective cohort study with electronic health record data from 120,000 inpatient admissions at an urban, academic center in Manhattan. The primary outcome was thirty-day readmission for three causes: all-cause, congestive heart failure, and chronic coronary atherosclerotic disease. Predictive modeling was performed via L1-regularized logistic regression. Calibration methods were compared including Platt Scaling, Logistic Calibration, and Prevalence Adjustment. Performance of predictive modeling and calibration was assessed via discrimination (c-statistic), calibration (Spiegelhalter Z-statistic, Root Mean Square Error [RMSE] of binned predictions, Sanders and Murphy Resolutions of the Brier Score, Calibration Slope and Intercept), and clinical usefulness (utility terms represented as costs). The amount of validation data necessary to apply each calibration algorithm was also assessed. Results C-statistics by diagnosis ranged from 0.7 for all-cause readmission to 0.86 (0.78–0.93) for congestive heart failure. Logistic Calibration and Platt Scaling performed best and this difference required analyzing multiple metrics of calibration simultaneously, in particular Calibration Slopes and Intercepts. Clinical usefulness analyses provided optimal risk thresholds, which varied by reason for readmission, outcome prevalence, and calibration algorithm. Utility analyses also suggested maximum tolerable intervention costs, e.g. $1,720 for all-cause readmissions based on a published cost of readmission of $11,000. Conclusions Choice of calibration method depends on availability of validation data and on performance. Improperly calibrated models may contribute to higher costs of intervention as measured via clinical usefulness. Decision-makers must understand underlying utilities or costs inherent in the use-case at hand to assess usefulness and will obtain the optimal risk threshold to trigger intervention and intervention cost limits as a result.

show abstract

Calibrating predictive model estimates to support personalized medicine

Cited by 150 publications

References 28 publications

icuARM-II

icuARM-II

Using Patients Like My Patient for Clinical Decision Support: Institution-Specific Probability of Celiac Disease Diagnosis Using Simplified Near-Neighbor Classification

Beyond discrimination: A comparison of calibration methods and clinical usefulness of predictive models of readmission risk

Contact Info

Product

Resources

About