Post-hoc Uncertainty Calibration for Domain Drift Scenarios

Tomani, Christian; Gruber, Sebastian; Erdem, Muhammed Ebrar; Cremers, Daniel; Buettner, Florian

doi:10.48550/arxiv.2012.10988

Cited by 3 publications

(4 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A method is considered to be calibrated (or reliable) if the confidence of predictions matches the probability of being correct for all confidence levels 22,45 . Formally, the empirical and predicted CDFs should be identical 46…”

Section: Definitions: Calibration and Sharpnessmentioning

confidence: 99%

“…Two major concepts resulting from these studies are calibration (reliability), and sharpness (resolution). A probabilistic prediction method is said to be calibrated if the confidence of predictions matches the probability of being correct for all confidence levels 45 . A calibrated method is sharp if it produces the tightest possible confidence intervals 46 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The long road to calibrated prediction uncertainty in computational chemistry

Pernot

2022

Preprint

View full text Add to dashboard Cite

a) Uncertainty quantification (UQ) in computational chemistry (CC) is still in its infancy.Very few CC methods are designed to provide a confidence level on their predictions, and most users still rely improperly on the mean absolute error as an accuracy metric. The development of reliable uncertainty quantification methods is essential, notably for computational chemistry to be used confidently in industrial processes. A review of the CC-UQ literature shows that there is no common standard procedure to report nor validate prediction uncertainty. I consider here analysis tools using concepts (calibration and sharpness) developed in meteorology and machine learning for the validation of probabilistic forecasters. These tools are adapted to CC-UQ and applied to datasets of prediction uncertainties provided by composite methods, Bayesian Ensembles methods, machine learning and a posteriori statistical methods.

show abstract

Section: Definitions: Calibration and Sharpnessmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The long road to calibrated prediction uncertainty in computational chemistry

Pernot

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Intervals-based testing. In the CS framework, a method is considered to be calibrated if the confidence of its predictions matches the probability of being correct for all confidence levels, 5,37 which can be reformulated as "prediction intervals should have the correct coverage". 21 It is convenient here to deal with prediction errors instead of predicted values, and one defines the prediction interval coverage probability (PICP) as 38…”

Section: Calibrationmentioning

confidence: 99%

Prediction uncertainty validation for computational chemists

Pernot

2022

Preprint

View full text Add to dashboard Cite

Validation of prediction uncertainty (PU) is becoming an essential tool for modern computational chemistry. Designed to quantify the reliability of predictions in meteorology, the calibration-sharpness (CS) framework is now widely used to optimize and validate uncertainty-aware machine learning (ML) methods. However, its application is not limited to ML and it can serve as a principled framework for any PU validation. The present article is intended as a step-by-step introduction to the concepts and techniques of PU validation in the CS framework, adapted to the specifics of computational chemistry. The presented methods range from elementary graphical checks to more sophisticated ones based on local calibration statistics. The concept of tightness, is introduced. The methods are illustrated on synthetic datasets and applied to uncertainty quantification data extracted from the computational chemistry literature.

show abstract

“…This is naturally a substantial problem in datasets with few samples and a high amount of features, such as image, language (with high dimensional embedding spaces) or long multivariate time series datasets. Nevertheless, this issue is distinct from the field of out-of-distribution (OOD) samples, where a rich body of research exists already [33,36,20,35,42,34]. The difference to in-domain scenarios is that OOD samples are considered to not be lying on the input data manifold.…”

Section: Introductionmentioning

confidence: 99%

CHALLENGER: Training with Attribution Maps

Tomani¹,

Cremers²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance. Regularization is key in deep learning, especially when training complex models on relatively small datasets. In order to understand inner workings of neural networks, attribution methods such as Layer-wise Relevance Propagation (LRP) have been extensively studied, particularly for interpreting the relevance of input features. We introduce Challenger, a module that leverages the explainable power of attribution maps in order to manipulate particularly relevant input patterns. Therefore, exposing and subsequently resolving regions of ambiguity towards separating classes on the ground-truth data manifold, an issue that arises particularly when training models on rather small datasets. Our Challenger module increases model performance through building more diverse filters within the network and can be applied to any input data domain. We demonstrate that our approach results in substantially better classification as well as calibration performance on datasets with only a few samples up to datasets with thousands of samples. In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.

show abstract

Post-hoc Uncertainty Calibration for Domain Drift Scenarios

Cited by 3 publications

References 9 publications

The long road to calibrated prediction uncertainty in computational chemistry

The long road to calibrated prediction uncertainty in computational chemistry

Prediction uncertainty validation for computational chemists

CHALLENGER: Training with Attribution Maps

Contact Info

Product

Resources

About