Evaluation of Uncertainty Quantification in Deep Learning

Ståhl, Niclas; Falkman, Göran; Karlsson, Alexander; Mathiason, Gunnar

doi:10.1007/978-3-030-50146-4_41

Cited by 33 publications

(17 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Future work will explore how the bias between high measurements and high uncertainty can be corrected for and may also consider alternative strategies which are known to produce substantially distinct estimates of uncertainty [30]. However, it is unclear whether Monte-Carlo techniques that employ dropout at test time [31] could reach sufficient predictive performance, whereas more faithful approximations of Bayesian inference with Markov chain Monte-Carlo [32] may not be computationally viable.…”

Section: Discussionmentioning

confidence: 99%

Uncertainty-Aware Body Composition Analysis with Deep Regression Ensembles on UK Biobank MRI

Langner¹,

Gustafsson²,

Avelin³

et al. 2021

Preprint

View full text Add to dashboard Cite

To enable fast and automated analysis of body composition from UK Biobank MRI with accurate estimates of individual measurement errors. Methods: In an ongoing large-scale imaging study the UK Biobank has acquired MRI of over 40,000 men and women aged 44-82. Phenotypes derived from these images, such as body composition, can reveal new links between genetics, cardiovascular disease, and metabolic conditions. In this retrospective study, neural networks were trained to provide six measurements of body composition from UK Biobank neck-to-knee body MRI. A ResNet50 architecture can automatically predict these values by image-based regression, but may also produce erroneous outliers. Predictive uncertainty, which could identify these failure cases, was therefore modeled with a mean-variance loss and ensembling. Its estimates of individual prediction errors were evaluated in cross-validation on over 8,000 subjects, tested on another 1,000 cases, and finally applied for inference. Results: Relative measurement errors below 5% were achieved on all but one target, for intra-class correlation coefficients (ICC) above 0.97 both in validation and testing. Both mean-variance loss and ensembling yielded improvements and provided uncertainty estimates that highlighted some of the worst outlier predictions. Combined, they reached the highest quality, but also exhibited a consistent bias towards high uncertainty in heavyweight subjects. Conclusion: Mean-variance regression and ensembling provided complementary benefits for automated body composition measurements from UK Biobank MRI, reaching high speed and accuracy. These values were inferred for the entire cohort, with uncertainty estimates that can approximate the measurement errors and identify some of the worst outliers automatically.

show abstract

Section: Discussionmentioning

confidence: 99%

Uncertainty-Aware Body Composition Analysis with Deep Regression Ensembles on UK Biobank MRI

Langner¹,

Gustafsson²,

Avelin³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Alternative methods are based, indicatively, on ensembles of NN optimization iterates or independently trained NNs [39][40][41][42][43][44][45][46][47][48][49][50], as well as on the evidential framework [51][52][53][54][55][56][57][58][59]. Although Bayesian methods and ensembles are thoroughly discussed in this paper, the interested reader is also directed to the recent review studies in [60][61][62][63][64][65][66][67][68][69][70][71][72] for more information. Clearly, in the context of SciML, which may involve differential equations with unknown or uncertain terms and parameters, UQ becomes an even more demanding task; see Fig.…”

Section: Motivation and Scope Of The Papermentioning

confidence: 99%

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. To demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed, which we will make available as open-source library of all codes included in this framework.

show abstract

“…As the variation among the data in aleatory uncertainty is often observable, we can well quantify the uncertainty and assess the risks. Quantification of epistemic uncertainty is more challenging because AI systems are forced to extrapolate over unseen situations (Ståhl, Falkman, Karlsson, & Mathiason, 2020). In the literature of uncertainty quantification, one of the most widely recognized techniques are prediction intervals (PI).…”

Section: Uncertainty Quantificationmentioning

confidence: 99%

Socially Responsible AI Algorithms: Issues, Purposes, and Challenges

Cheng

Varshney

Liu

2021

jair

104

View full text Add to dashboard Cite

In the current era, people and society have grown increasingly reliant on artificial intelligence (AI) technologies. AI has the potential to drive us towards a future in which all of humanity flourishes. It also comes with substantial risks for oppression and calamity. Discussions about whether we should (re)trust AI have repeatedly emerged in recent years and in many quarters, including industry, academia, healthcare, services, and so on. Technologists and AI researchers have a responsibility to develop trustworthy AI systems. They have responded with great effort to design more responsible AI algorithms. However, existing technical solutions are narrow in scope and have been primarily directed towards algorithms for scoring or classification tasks, with an emphasis on fairness and unwanted bias. To build long-lasting trust between AI and human beings, we argue that the key is to think beyond algorithmic fairness and connect major aspects of AI that potentially cause AI’s indifferent behavior. In this survey, we provide a systematic framework of Socially Responsible AI Algorithms that aims to examine the subjects of AI indifference and the need for socially responsible AI algorithms, define the objectives, and introduce the means by which we may achieve these objectives. We further discuss how to leverage this framework to improve societal well-being through protection, information, and prevention/mitigation. This article appears in the special track on AI & Society.

show abstract

Evaluation of Uncertainty Quantification in Deep Learning

Cited by 33 publications

References 14 publications

Uncertainty-Aware Body Composition Analysis with Deep Regression Ensembles on UK Biobank MRI

Uncertainty-Aware Body Composition Analysis with Deep Regression Ensembles on UK Biobank MRI

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Socially Responsible AI Algorithms: Issues, Purposes, and Challenges

Contact Info

Product

Resources

About