Well-calibrated and specialized probability estimation trees

Abstract

In many predictive modeling scenarios, the production set inputs that later will be used for the actual prediction is available and could be utilized in the modeling process. In fact, many predictive models are generated with an existing production set in mind. Despite this, few approaches utilize this information in order to produce models optimized on the production set at hand. If these models need to be comprehensible, the oracle coaching framework can be applied, often resulting in interpretable models, e.g., decision trees and rule sets, with accuracies on par with opaque models like neural networks and ensembles, on the specific production set. In oracle coaching, a strong but opaque predictive model is used to label instances, including the production set, which are later learned by a weaker but interpretable model. In this paper, oracle coaching is, for the first time, used for improving the calibration of probabilistic predictors. More specifically, setups where oracle coaching are combined with the techniques Platt scaling, isotonic regression and Venn-Abers are suggested and evaluated for calibrating probability estimation trees (PETs). A key contribution is the setup designs ensuring that the oracle-coached PETs, that per definition utilize knowledge about production data, remain well-calibrated. In the experimentation, using 23 publicly available data sets, it is shown that oracle-coached models are not only more accurate, but also significantly better calibrated, compared to standard induction. Interestingly enough, this holds both for the uncalibrated PETs, and for all calibration techniques evaluated, i.e., Platt scaling, isotonic regression and Venn-Abers. As expected, all three external techniques significantly improved the calibration of the original PETs. Finally, an outright comparison between the three external calibration techniques showed that Venn-Abers significantly outperformed the alternatives in most setups.

Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods

“…The idea of scaling or calibration methods is to turn such scores into proper, well-calibrated probabilities, that is, to learn a mapping from scores to the unit interval that can be applied to the output of a predictor as a kind of post-processing step (Flach, 2017). Examples of such methods include binning (Zadrozny and Elkan, 2001), isotonic regression (Zadrozny and Elkan, 2002), logistic scaling (Platt, 1999) and improvements thereof (Kull et al, 2017), as well as the use of Venn predictors (Johansson et al, 2018).…”

Investigating the impact of calibration on the quality of explanations

“…In [25], the authors focus on the size of the prediction interval, where a smaller (tighter) interval is seen as more informative. The same authors further develop the implementation of CP with decision trees in [29], where they introduce CP with oracle coaching for improving the calibration of probabilistic prediction.…”

Classifier Calibration: How to assess and improve predicted class probabilities: a survey

“…Multi-class probabilistic outputs can be obtained by combining multiple binary classifiers or calibration methods (Dietterich & Bakiri, 1994;Johansson et al, 2018;Leathart, Frank, Pfahringer, & Holmes, 2019;Wu, Lin, & Weng, 2004), and calibration can be performed within these combination approaches. However, it is often desirable to have a native multi-class model, where the level of calibration can Fig.…”