Classifier Calibration: How to assess and improve predicted class probabilities: a survey

Abstract

This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main

Multiclass Segmentation of Breast Tissue and Suspicious Findings: A Simulation-Based Study for the Development of Self-Steering Tomosynthesis

“…Imbalanced class data can result in models for which the overall performance is not representative of the performance for the underrepresented classes. Post hoc Dirichlet calibration (DC) was used to adjust the model’s output probabilities and address potential problems with overconfidence in the predictions [ 13 ].…”

“…Probability calibration can be used to reduce the false positive rates of multiclass classifiers. In accordance with the main concept of calibration [ 13 ], a multiclass probabilistic classifier should only be considered well-calibrated if instances of a particular class receive probabilities in accordance with the actual class distribution of the data. For example, if we have amongst the test instances a predicted probability vector s = [0.1, 0.2, 0.7], the class distribution of s should be approximately 10%, 20%, and 70% for the first, second, and third classes, respectively.…”

“…For example, if we have amongst the test instances a predicted probability vector s = [0.1, 0.2, 0.7], the class distribution of s should be approximately 10%, 20%, and 70% for the first, second, and third classes, respectively. This is a fundamental property when using a classifier for cost-sensitive classification or for human decision making, because a calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its predictions [ 13 ].…”

Exploration of Production Data for Predictive Maintenance of Industrial Equipment: A Case Study

“…where w ∈ R is the shape parameter and b ∈ R is the location parameter. The parameters are estimated by maximizing the log-likelihood on the validation set [32], [33].…”