Depth Uncertainty in Neural Networks

Antorán, Javier; Allingham, James Urquhart; Hernández-Lobato, José Miguel

doi:10.48550/arxiv.2006.08437

Cited by 7 publications

(13 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Later, Gal and Ghahramani [12] showed that dropout can be applied at test-time, called Monte Carlo (MC) dropout, which can be viewed as an approximate Bayesian technique and yields better estimates of uncertainty. Several improvements have also been proposed to improve MC dropout [1,4,13,26,40,51]. Unlike MC dropout which generates random masks on the fly at every iteration, we demonstrate that pruning-derived dropout masks can lead to significantly better performance.…”

Section: Introductionmentioning

confidence: 86%

“…The resulting algorithm enables us to obtain a diverse ensemble of non-overlapping subnetworks within one deep neural network. That is, we are able create many models out of one 1 . We experimentally demonstrate that subnetwork 2 Related Works Ensemble Learning An ensemble of models has long known to be an effective method to boost the performance of machine learning models [7,52].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Ex uno plures: Splitting One Model into an Ensemble of Subnetworks

Zhang,

Gao,

Sabuncu

2021

Preprint

View full text Add to dashboard Cite

Monte Carlo (MC) dropout [12] is a simple and efficient ensembling method that can improve the accuracy and confidence calibration of high-capacity deep neural network models. However, MC dropout is not as effective as more computeintensive methods such as deep ensembles [30]. This performance gap can be attributed to the relatively poor quality of individual models in the MC dropout ensemble and their lack of diversity. These issues can in turn be traced back to the coupled training and substantial parameter sharing of the dropout models. Motivated by this perspective, we propose a strategy to compute an ensemble of subnetworks, each corresponding to a non-overlapping dropout mask computed via a pruning strategy and trained independently. We show that the proposed subnetwork ensembling method can perform as well as standard deep ensembles in both accuracy and uncertainty estimates, yet with a computational efficiency similar to MC dropout. Lastly, using several computer vision datasets like CIFAR10/100, CUB200, and Tiny-Imagenet, we experimentally demonstrate that subnetwork ensembling also consistently outperforms recently proposed approaches that efficiently ensemble neural networks.

show abstract

Section: Introductionmentioning

confidence: 86%

Section: Introductionmentioning

confidence: 99%

Ex uno plures: Splitting One Model into an Ensemble of Subnetworks

Zhang,

Gao,

Sabuncu

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…A more tractable model class are deep ensemble methods [35,9,13,12], although they are computationally still expensive. There are however some ideas to make them less expensive by distilling their uncertainties into simpler models [39,55,26,3].…”

Section: Related Workmentioning

confidence: 99%

Deep Classifiers with Label Noise Modeling and Distance Awareness

Fortuin¹,

Collier²,

Wenzel³

et al. 2021

Preprint

View full text Add to dashboard Cite

Uncertainty estimation in deep learning has recently emerged as a crucial area of interest to advance reliability and robustness in safety-critical applications. While there have been many proposed methods that either focus on distance-aware model uncertainties for out-of-distribution detection or on input-dependent label uncertainties for in-distribution calibration, both of these types of uncertainty are often necessary. In this work, we propose the HetSNGP method for jointly modeling the model and data uncertainty. We show that our proposed model affords a favorable combination between these two complementary types of uncertainty and thus outperforms the baseline methods on some challenging out-of-distribution datasets, including CIFAR-100C, Imagenet-C, and Imagenet-A. Moreover, we propose HetSNGP Ensemble, an ensembled version of our method which provides an additional type of uncertainty and also outperforms other ensemble baselines.

show abstract

“…Recently, calibration has been rediscovered with the deep learning booming trend. Indeed, deep learning models are no exception to the rule and are even more than traditional methods subject to model uncertainty due to their complex architectures [5]. That's why measuring calibration error and develop calibration techniques for deep neural networks have become an important research topic these recent years [44,26,47].…”

Section: Literature Reviewmentioning

confidence: 99%

Credit scoring using neural networks and SURE posterior probability calibration

Garcin,

Stéphan

2021

Preprint

View full text Add to dashboard Cite

In this article we compare the performances of a logistic regression and a feed forward neural network for credit scoring purposes. Our results show that the logistic regression gives quite good results on the dataset and the neural network can improve a little the performance. We also consider different sets of features in order to assess their importance in terms of prediction accuracy. We found that temporal features (i.e. repeated measures over time) can be an important source of information resulting in an increase in the overall model accuracy. Finally, we introduce a new technique for the calibration of predicted probabilities based on Stein's unbiased risk estimate (SURE). This calibration technique can be applied to very general calibration functions. In particular, we detail this method for the sigmoid function as well as for the Kumaraswamy function, which includes the identity as a particular case. We show that stacking the SURE calibration technique with the classical Platt method can improve the calibration of predicted probabilities.

show abstract

Depth Uncertainty in Neural Networks

Cited by 7 publications

References 11 publications

Ex uno plures: Splitting One Model into an Ensemble of Subnetworks

Ex uno plures: Splitting One Model into an Ensemble of Subnetworks

Deep Classifiers with Label Noise Modeling and Distance Awareness

Credit scoring using neural networks and SURE posterior probability calibration

Contact Info

Product

Resources

About