Calibration tests in multi-class classification: A unifying framework

Widmann, David; Lindsten, Fredrik; Zachariah, Dave

doi:10.48550/arxiv.1910.11385

Cited by 4 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As calibration fidelity measurements using reliability diagrams are originally designed for binary classifiers, there have been numerous proposals for their extensions to multi-class classifiers [37][38][39]. We consider two relatively simple methods; the first is a standard used in Guo et al [37], where only the predicted probability for the most confident prediction of each sample is used to plot the reliability diagram.…”

Section: Evaluating Uncertainty Quantification Methodsmentioning

confidence: 99%

Deep neural network uncertainty quantification for LArTPC reconstruction

Koh,

Mishra,

Terao

2023

J. Inst.

View full text Add to dashboard Cite

We evaluate uncertainty quantification (UQ) methods for deep learning applied to liquid argon time projection chamber (LArTPC) physics analysis tasks. As deep learning applications enter widespread usage among physics data analysis, neural networks with reliable estimates of prediction uncertainty and robust performance against overconfidence and out-of-distribution (OOD) samples are critical for their full deployment in analyzing experimental data. While numerous UQ methods have been tested on simple datasets, performance evaluations for more complex tasks and datasets are scarce. We assess the application of selected deep learning UQ methods on the task of particle classification using the PiLArNet [1] monte carlo 3D LArTPC point cloud dataset. We observe that UQ methods not only allow for better rejection of prediction mistakes and OOD detection, but also generally achieve higher overall accuracy across different task settings. We assess the precision of uncertainty quantification using different evaluation metrics, such as distributional separation of prediction entropy across correctly and incorrectly identified samples, receiver operating characteristic curves (ROCs), and expected calibration error from observed empirical accuracy. We conclude that ensembling methods can obtain well calibrated classification probabilities and generally perform better than other existing methods in deep learning UQ literature.

show abstract

Section: Evaluating Uncertainty Quantification Methodsmentioning

confidence: 99%

Deep neural network uncertainty quantification for LArTPC reconstruction

Koh,

Mishra,

Terao

2023

J. Inst.

View full text Add to dashboard Cite

show abstract

“…Our calibration technique is independent of the binning scheme/bins. This is important, because as [53] and [26] have also highlighted, binning scheme leads to underestimated calibration errors. We name our loss function, Multiclass Difference of Confidence and Accuracy (MDCA), and apply it for each mini-batch during training.…”

Section: Proposed Auxiliary Loss: Mdcamentioning

confidence: 99%

A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration

Hebbalaguppe¹,

Prakash²,

Madan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) are known to make overconfident mistakes, which makes their use problematic in safety-critical applications. State-of-the-art (SOTA) calibration techniques improve on the confidence of predicted labels alone, and leave the confidence of non-max classes (e.g. top-2, top-5) uncalibrated. Such calibration is not suitable for § Equal contribution label refinement using post-processing. Further, most SOTA techniques learn a few hyper-parameters post-hoc, leaving out the scope for image, or pixel specific calibration. This makes them unsuitable for calibration under domain shift, or for dense prediction tasks like semantic segmentation. In this paper, we argue for intervening at the train time itself, so as to directly produce calibrated DNN models. We propose a novel auxiliary loss function: Multi-class Difference in Confidence and Accuracy (MDCA), to achieve the same.

show abstract

“…the support of p), the necessary sample complexity grows linearly in the support of p. Of course, for a trivial predictors that map all inputs x to the same prediction q 0 (i.e. p(x) = q 0 , ∀x ∈ X ) distribution calibration is easy to certify (Widmann et al, 2019), but such predictors have no practical use.…”

Section: Decision Calibration Generalizes Existing Notions Of Calibra...mentioning

confidence: 99%

“…Weaker notions such as confidence (weak) calibration (Platt et al, 1999;Guo et al, 2017), class-wise calibration or average calibration average calibration (Kuleshov et al, 2018) are more commonly used in practice. To unify these notions, (Widmann et al, 2019) proposes F-calibration but lacks detailed guidance on which notions to use.…”

Section: Related Workmentioning

confidence: 99%

Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration

Zhao

Kim

Sahoo

et al. 2021

Preprint

View full text Add to dashboard Cite

When facing uncertainty, decision-makers want predictions they can trust. A machine learning provider can convey confidence to decision-makers by guaranteeing their predictions are distribution calibrated-amongst the inputs that receive a predicted class probabilities vector q, the actual distribution over classes is q. For multi-class prediction problems, however, achieving distribution calibration tends to be infeasible, requiring sample complexity exponential in the number of classes C. In this work, we introduce a new notion-decision calibration-that requires the predicted distribution and true distribution to be "indistinguishable" to a set of downstream decision-makers. When all possible decision makers are under consideration, decision calibration is the same as distribution calibration. However, when we only consider decision makers choosing between a bounded number of actions (e.g. polynomial in C), our main result shows that decisions calibration becomes feasible-we design a recalibration algorithms that requires sample complexity polynomial in the number of actions and the number of classes. We validate our recalibration algorithm empirically: compared to existing methods, decision calibration improves decision-making on skin lesion and ImageNet classification with modern neural network predictors.

show abstract

Calibration tests in multi-class classification: A unifying framework

Cited by 4 publications

References 12 publications

Deep neural network uncertainty quantification for LArTPC reconstruction

Deep neural network uncertainty quantification for LArTPC reconstruction

A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration

Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration

Contact Info

Product

Resources

About