Accurate Uncertainty Estimation and Decomposition in Ensemble Learning

Liu, Jeremiah Zhe; Paisley, John; Kioumourtzoglou, Marianthi-Anna; Coull, Brent A.

doi:10.48550/arxiv.1911.04061

Cited by 3 publications

(3 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…And J.Z. Liu, Paisley, Kioumourtzoglou, and Coull (2019) adopt a calibration procedure to achieve better posterior distributions for a Bayesian non-parametric ensemble model.…”

Section: Related Workmentioning

confidence: 99%

Classifier Calibration: How to assess and improve predicted class probabilities: a survey

Filho¹,

Song²,

Nieto³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main

show abstract

“…And J.Z. Liu, Paisley, Kioumourtzoglou, and Coull (2019) adopt a calibration procedure to achieve better posterior distributions for a Bayesian non-parametric ensemble model.…”

Section: Related Workmentioning

confidence: 99%

Classifier Calibration: How to assess and improve predicted class probabilities: a survey

Filho¹,

Song²,

Nieto³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In this work, we enable adaptation to distinct real masks by proposing a variational approach to estimate mask uncertainty. Popular uncertainty estimation methods include deep ensemble [6,19,21], i.e., training a group of networks from random weight initialization, which brings promising performance and high-quality uncertainty quantification. Bayesian neural networks (BNN) [2] forms another popular stream.…”

Section: Related Workmentioning

confidence: 99%

Modeling Mask Uncertainty in Hyperspectral Image Reconstruction

Wang¹,

Zhang²,

Yuan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, hyperspectral imaging (HSI) has attracted increasing research attention, especially for the ones based on a coded aperture snapshot spectral imaging (CASSI) system. Existing deep HSI reconstruction models are generally trained on paired data to retrieve original signals upon 2D compressed measurements given by a particular optical hardware mask in CASSI, during which the mask largely impacts the reconstruction performance and could work as a "model hyperparameter" governing on data augmentations. This mask-specific training style will lead to a hardware miscalibration issue, which sets up barriers to deploying deep HSI models among different hardware and noisy environments. To address this challenge, we introduce mask uncertainty for HSI with a complete variational Bayesian learning treatment and explicitly model it through a mask decomposition inspired by real hardware. Specifically, we propose a novel Graph-based Self-Tuning (GST) network to reason uncertainties adapting to varying spatial structures of masks among different hardware. Moreover, we develop a bilevel optimization framework to balance HSI reconstruction and uncertainty estimation, accounting for the hyperparameter property of masks. Extensive experimental results and model discussions validate the effectiveness (over 33/30 dB) of the proposed GST method under two miscalibration scenarios and demonstrate a highly competitive performance compared with the state-of-the-art well-calibrated methods. Our code and pre-trained model are available at https://github.com/Jiamian-Wang/mask_uncertainty_spectral_SCI

show abstract

“…Providing a prediction without the associated uncertainty can be dangerous in cases where the prediction is subsequently used to make important decisions [5]. The sources of uncertainty are often decomposed into two parts-the aleatoric and the epistemic [6]. The first is inherent in the process under study, while the second depends on inadequate knowledge of the model that is most suited to explaining the data.…”

Section: Introductionmentioning

confidence: 99%

Optimization under Uncertainty: Machine Learning Approach

Mohammadi¹,

Farsijani²

2023

IJIMES

View full text Add to dashboard Cite

Data is the new oil. From the beginning of the 21st century, data is similar to what oil was in the 18th century, an immensely untapped valuable asset. This paper reviews recent advances in the field of optimization under uncertainty via a modern data lens and highlights key research challenges and the promise of data-driven optimization that organically integrates machine learning and mathematical programming for decision-making under uncertainty. A brief review of classical mathematical programming techniques for hedging against uncertainty is first presented, along with their wide spectrum of applications in Process Systems Engineering. we provide an introduction to the topic of uncertainty in machine learning as well as an overview of attempts so far at handling uncertainty in general and formalizing this distinction in particular. In line with the statistical tradition, uncertainty has long been perceived as almost synonymous with standard probability and probabilistic predictions. Yet, due to the steadily increasing relevance of machine learning for practical applications and related issues such as safety requirements, new problems, and challenges have recently been identified by machine learning scholars, and these problems may call for new methodological developments.

show abstract

Accurate Uncertainty Estimation and Decomposition in Ensemble Learning

Cited by 3 publications

References 35 publications

Classifier Calibration: How to assess and improve predicted class probabilities: a survey

Classifier Calibration: How to assess and improve predicted class probabilities: a survey

Modeling Mask Uncertainty in Hyperspectral Image Reconstruction

Optimization under Uncertainty: Machine Learning Approach

Contact Info

Product

Resources

About