CaliForest

Abstract

Real-world predictive models in healthcare should be evaluated in terms of discrimination, the ability to differentiate between high and low risk events, and calibration, or the accuracy of the risk estimates. Unfortunately, calibration is often neglected and only discrimination is analyzed. Calibration is crucial for personalized medicine as they play an increasing role in the decision making process. Since random forest is a popular model for many healthcare applications, we propose CaliForest, a new calibrated random forest. Unlike existing calibration methodologies, CaliForest utilizes the out-of-bag samples to avoid the explicit construction of a calibration set. We evaluated CaliForest on two risk prediction tasks obtained from the publicly-available MIMIC-III database. Evaluation on these binary prediction tasks demonstrates that CaliForest can achieve the same discriminative power as random forest while obtaining a better-calibrated model evaluated across six different metrics. CaliForest is published on the standard Python software repository and the code is openly available on Github. CCS CONCEPTS• Applied computing → Health informatics; • General and reference → Empirical studies; • Computing methodologies → Classification and regression trees; Bagging.

Machine learning for the life-time risk prediction of Alzheimer’s disease: a systematic review

“…With the potential of causing death due to incorrect treatment in the most serious of circumstances. 82 Therefore, we recommend that authors aim to produce highly calibrated models and also report calibration statistics.…”