Deep Ensembles from a Bayesian Perspective

Hoffmann, Lara; Elster, Clemens

doi:10.48550/arxiv.2105.13283

Cited by 12 publications

(14 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The advantage of BMA is the ability to simultaneously consider all possible predictions given a prior and conditioned on training data, thereby mitigating the risk inherent in trusting the predictions of any single model. A line of recent work considers individual deep ensemble members as samples from p(f |D), and treats the ensemble output as a Monte Carlo approximation to BMA (Hoffmann & Elster, 2021;Wilson & Izmailov, 2020). We also note a line work that emphasizes differences between ensembling and BMA (Minka, 2000;He et al, 2020).…”

Section: Interpretationsmentioning

confidence: 99%

Deep Ensembles Work, But Are They Necessary?

Abe¹,

Buchanan²,

Pleiss³

et al. 2022

Preprint

View full text Add to dashboard Cite

Ensembling neural networks is an effective way to increase accuracy, and can often match the performance of larger models. This observation poses a natural question: given the choice between a deep ensemble and a single neural network with similar accuracy, is one preferable over the other? Recent work suggests that deep ensembles may offer benefits beyond predictive power: namely, uncertainty quantification and robustness to dataset shift. In this work, we demonstrate limitations to these purported benefits, and show that a single (but larger) neural network can replicate these qualities. First, we show that ensemble diversity, by any metric, does not meaningfully contribute to an ensemble's ability to detect out-of-distribution (OOD) data, and that one can estimate ensemble diversity by measuring the relative improvement of a single larger model. Second, we show that the OOD performance afforded by ensembles is strongly determined by their in-distribution (InD) performance, and-in this sense-is not indicative of any "effective robustness." While deep ensembles are a practical way to achieve performance improvement (in agreement with prior work), our results show that they may be a tool of convenience rather than a fundamentally better model class.

show abstract

Section: Interpretationsmentioning

confidence: 99%

Deep Ensembles Work, But Are They Necessary?

Abe¹,

Buchanan²,

Pleiss³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In contrast to the bagging algorithms discussed above, heterogeneity of deep ensemble members does not stem from bootstrapping the data, but rather from stochasticity in initializing and optimizing the neural networks. Several contributions suggest that deep ensembles benefit prediction performance, uncertainty quantification and out-of-distribution generalization (Fort et al, 2019;Wilson and Izmailov, 2020;Hoffmann and Elster, 2021). Abe et al (2022) question the extent to which these benefits hold.…”

Section: Classical Deep Ensembles Classical Deep Ensembles Combine Pr...mentioning

confidence: 99%

Deep interpretable ensembles

Kook¹,

Götschi²,

Baumann³

et al. 2022

Preprint

View full text Add to dashboard Cite

Ensembles improve prediction performance and allow uncertainty quantification by aggregating predictions from multiple models. In deep ensembling, the individual models are usually black box neural networks, or recently, partially interpretable semi-structured deep transformation models. However, interpretability of the ensemble members is generally lost upon aggregation. This is a crucial drawback of deep ensembles in high-stake decision fields, in which interpretable models are desired. We propose a novel transformation ensemble which aggregates probabilistic predictions with the guarantee to preserve interpretability and yield uniformly better predictions than the ensemble members on average. Transformation ensembles are tailored towards interpretable deep transformation models but are applicable to a wider range of probabilistic neural networks. In experiments on several publicly available data sets, we demonstrate that transformation ensembles perform on par with classical deep ensembles in terms of prediction performance, discrimination, and calibration. In addition, we demonstrate how transformation ensembles quantify both aleatoric and epistemic uncertainty, and produce minimax optimal predictions under certain conditions.

show abstract

“…In addition to approaches based on variational inference, there are other approaches to quantify the uncertainty about θ [18,[43][44][45]. The approach of deep ensembles [18], introduced as a sort of frequentist approach, can also be viewed as a Bayesian approach [46,47].…”

Section: A Primer On (Deep) Regression and Uncertaintymentioning

confidence: 99%

A framework for benchmarking uncertainty in deep regression

2022

Self Cite

View full text Add to dashboard Cite

We propose a framework for the assessment of uncertainty quantification in deep regression. The framework is based on regression problems where the regression function is a linear combination of nonlinear functions. Basically, any level of complexity can be realized through the choice of the nonlinear functions and the dimensionality of their domain. Results of an uncertainty quantification for deep regression are compared against those obtained by a statistical reference method. The reference method utilizes knowledge about the underlying nonlinear functions and is based on Bayesian linear regression using a prior reference. The flexibility, together with the availability of a reference solution, makes the framework suitable for defining benchmark sets for uncertainty quantification. Reliability of uncertainty quantification is assessed in terms of coverage probabilities, and accuracy through the size of calculated uncertainties. We illustrate the proposed framework by applying it to current approaches for uncertainty quantification in deep regression. In addition, results for three real-world regression tasks are presented.

show abstract

Deep Ensembles from a Bayesian Perspective

Cited by 12 publications

References 28 publications

Deep Ensembles Work, But Are They Necessary?

Deep Ensembles Work, But Are They Necessary?

Deep interpretable ensembles

A framework for benchmarking uncertainty in deep regression

Contact Info

Product

Resources

About