Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification

Chung, Youngseog; Char, Ian; Guo, Han; Schneider, Jeff; Neiswanger, Willie

doi:10.48550/arxiv.2109.10254

Cited by 23 publications

(34 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each of the metrics presented in this section may correspond to first-or second-order statistics of the predictive model, or to the whole distribution (PDF or CDF). More information on evaluation metrics as well as comparative studies can be found in [107][108][109][110][111][112][113][114][115] and Appendix C.…”

Section: Evaluation: Accuracy and Uncertainty Quality Evaluationmentioning

confidence: 99%

“…These approaches can also prove useful for addressing model misspecification cases, e.g., when a wrong data noise model has been utilized (see also Section A.4). More information on the active area of research related to calibration approaches for function approximation can be found in [50,109,114,[117][118][119][120][121] and Appendix C. Computational results can be found in Sections 6.1, 6.3, and 6.5.…”

Section: Post-training Improvement: Calibrationmentioning

confidence: 99%

“…For GAN-FP, we used our own implementation following Section B.4. For post-training calibration, we used the package of Chung et al [114]. In the near future, we will provide additional implementation details in an upcoming manual and release our code in a public GitHub repository.…”

Section: Performance Versus Costmentioning

confidence: 99%

“…For example, isotonic regression, which is suitable for monotonic function fitting, can be used; an open-source implementation can be found in [114]. Note that if the model is calibrated, the dataset of Eq.…”

Section: C2 Post-training Calibration C21 Cdf Modification By Fitting...mentioning

confidence: 99%

See 3 more Smart Citations

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. To demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed, which we will make available as open-source library of all codes included in this framework.

show abstract

Section: Evaluation: Accuracy and Uncertainty Quality Evaluationmentioning

confidence: 99%

Section: Post-training Improvement: Calibrationmentioning

confidence: 99%

Section: Performance Versus Costmentioning

confidence: 99%

Section: C2 Post-training Calibration C21 Cdf Modification By Fitting...mentioning

confidence: 99%

See 2 more Smart Citations

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This definition is similar to the definitions based on credible intervals that have been used in many prior works [26]. However, it has been suggested that this definition should be referred to as average calibration because it reflects the average calibration over the entire dataset rather than calibration on each individual data point [27], [28]. It is possible to have uninformative models with perfect average calibration.…”

Section: Uncertainty In Deep Learning-based Regressionmentioning

confidence: 99%

Calibrated Diffusion Tensor Estimation

Karimi¹,

Warfield²,

Gholipour³

2021

Preprint

View full text Add to dashboard Cite

It is highly desirable to know how uncertain a model's predictions are, especially for models that are complex and hard to understand as in deep learning. Although there has been a growing interest in using deep learning methods in diffusion-weighted MRI, prior works have not addressed the issue of model uncertainty. Here, we propose a deep learning method to estimate the diffusion tensor and compute the estimation uncertainty. Data-dependent uncertainty is computed directly by the network and learned via loss attenuation. Model uncertainty is computed using Monte Carlo dropout. We also propose a new method for evaluating the quality of predicted uncertainties. We compare the new method with the standard least-squares tensor estimation and bootstrap-based uncertainty computation techniques. Our experiments show that when the number of measurements is small the deep learning method is more accurate and its uncertainty predictions are better calibrated than the standard methods. We show that the estimation uncertainties computed by the new method can highlight the model's biases, detect domain shift, and reflect the strength of noise in the measurements. Our study shows the importance and practical value of modeling prediction uncertainties in deep learning-based diffusion MRI analysis.

show abstract

Leveraging Uncertainty Estimates to Improve Segmentation Performance in Cardiac MR

Arega

Bricq

Mériaudeau

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In medical image segmentation, several studies have used Bayesian neural networks to segment and quantify the uncertainty of the images. These studies show that there might be an increased epistemic uncertainty in areas where there are semantically and visually challenging pixels. The uncertain areas of the image can be of a great interest as they can possibly indicate the regions of incorrect segmentation. To leverage the uncertainty information, we propose a segmentation model that incorporates the uncertainty into its learning process. Firstly, we generate the uncertainty estimate (sample variance) using Monte-Carlo dropout during training. Then we incorporate it into the loss function to improve the segmentation accuracy and probability calibration. The proposed method is validated on the publicly available EMIDEC MIC-CAI 2020 dataset that mainly focuses on segmentation of healthy and infarcted myocardium. Our method achieves the state of the art results outperforming the top ranked methods of the challenge. The experimental results show that adding the uncertainty information to the loss function improves the segmentation results by enhancing the geometrical and clinical segmentation metrics of both the scar and myocardium. These improvements are particularly significant at the visually challenging and difficult images which have higher epistemic uncertainty. The proposed system also produces more calibrated probabilities.

show abstract

Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification

Cited by 23 publications

References 15 publications

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Calibrated Diffusion Tensor Estimation

Leveraging Uncertainty Estimates to Improve Segmentation Performance in Cardiac MR

Contact Info

Product

Resources

About