Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

Zaidi, Sheheryar; Zela, Arbër; Elsken, Thomas; Holmes, Chris; Hutter, Frank; Teh, Yee Whye

doi:10.48550/arxiv.2006.08573

Cited by 3 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ensembling techniques constructed using diverse individual architectures have better generalization ability [30,33,46,47]. To our knowledge, this is the first study to leverage the different robustness properties of specific V1 neuronal populations to create diverse members of an ensemble.…”

Section: Discussionmentioning

confidence: 98%

“…The Variants Ensemble not only outperformed all the variants but also performed on par with ResNet18 on clean images (Fig. While diversity in the members of an ensemble has been found to be important to its generalization ability [30,33,46,47], ensembles of networks with the same architecture that only differ in their We also compared Variants Ensemble to a popular defense method that uses Gaussian Noise Training (GNT) as data augmentation [11]. We trained both ResNet18 (ResNet18-GNT) and standard VOneResNet18 (VOneResNet18-GNT) with GNT, observing an increased robustness for noise and blur categories (Fig.…”

Section: Ensemble Of Different Vonenet Variants Eliminates Robustness...mentioning

confidence: 98%

See 1 more Smart Citation

Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs

Baidya¹,

Dapello²,

DiCarlo³

et al. 2021

Preprint

View full text Add to dashboard Cite

While some convolutional neural networks (CNNs) have surpassed human visual abilities in object classification, they often struggle to recognize objects in images corrupted with different types of common noise patterns, highlighting a major limitation of this family of models. Recently, it has been shown that simulating a primary visual cortex (V1) at the front of CNNs leads to small improvements in robustness to these image perturbations. In this study, we start with the observation that different variants of the V1 model show gains for specific corruption types. We then build a new model using an ensembling technique, which combines multiple individual models with different V1 front-end variants. The model ensemble leverages the strengths of each individual model, leading to significant improvements in robustness across all corruption categories and outperforming the base model by 38% on average. Finally, we show that using distillation, it is possible to partially compress the knowledge in the ensemble model into a single model with a V1 front-end. While the ensembling and distillation techniques used here are hardly biologically-plausible, the results presented here demonstrate that by combining the specific strengths of different neuronal circuits in V1 it is possible to improve the robustness of CNNs for a wide range of perturbations.Recently, Dapello, Marques et al. observed that models that were more robust to white box attacks also had early stages that better predicted neuronal responses in the macaque primary visual cortex (V1) [12]. Inspired by this, the authors developed a novel class of hybrid CNNs, containing a model of primate V1 as the front-end, followed by a trainable standard CNN architecture back-end. This new model family, the VOneNet, was substantially more robust to adversarial attacks than the Preprint. Under review.

show abstract

Section: Discussionmentioning

confidence: 98%

Section: Ensemble Of Different Vonenet Variants Eliminates Robustness...mentioning

confidence: 98%

Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs

Baidya¹,

Dapello²,

DiCarlo³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Neural Ensemble Search [63] [11] is a similar approach with the key idea of gradual removal of the least promising operations from the supernetwork. Neural Ensemble Search via Sampling (NESS) [47] is supernetbased but does not require the ensemble models to have the same first layers.…”

Section: Search For Architectures Of Neural Ensemblesmentioning

confidence: 99%

“…It has been shown to work well if the models' mistakes are independent [21], which is helped by the models being different from each other [57]. NAS has been used [11,63] to produce models that together make a good ensemble. Modern supernetwork-based approaches seem very fitting to this purpose because they do not incur additional training costs for ensembles of arbitrarily large size (once a supernetwork is trained, weights for trained subnetworks can be extracted from it and used without additional retraining 1 ).…”

Section: Introdutionmentioning

confidence: 99%

Evolutionary Neural Cascade Search across Supernetworks

Chebykin,

Alderliesten,

Bosman

2022

Preprint

View full text Add to dashboard Cite

To achieve excellent performance with modern neural networks, having the right network architecture is important. Neural Architecture Search (NAS) concerns the automatic discovery of task-specific network architectures. Modern NAS approaches leverage supernetworks whose subnetworks encode candidate neural network architectures. These subnetworks can be trained simultaneously, 1 This holds for modern state-of-the-art approaches [8,46,55] but does not hold for all, especially older, approaches [30,49].

show abstract

“…Although Wenzel et al (2021) attempt to relax this issue by allowing more diversity in the ensemble, they vary just two hyperparameters. Similarly, Zaidi et al (2020) varied the architecture with fixed trainable hyperparameters to increase the ensemble diversity. By constructing diverse DNNs models through a methodical and automated approach, we hypothesize that the assumption of, and the eventual collapse, to one hypothesis can be avoided, thus providing robust and efficient estimates of uncertainty.…”

Section: Introductionmentioning

confidence: 99%

AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification

Égelé

Maulik

Krishnan

et al. 2022

2022 26th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

Deep neural networks are powerful predictors for a variety of tasks. However, they do not capture uncertainty directly. Using neural network ensembles to quantify uncertainty is competitive with approaches based on Bayesian neural networks while benefiting from better computational scalability. However, building ensembles of neural networks is a challenging task because, in addition to choosing the right neural architecture or hyperparameters for each member of the ensemble, there is an added cost of training each model. We propose Au-toDEUQ, an automated approach for generating an ensemble of deep neural networks. Our approach leverages joint neural architecture and hyperparameter search to generate ensembles. We use the law of total variance to decompose the predictive variance of deep ensembles into aleatoric (data) and epistemic (model) uncertainties. We show that AutoDEUQ outperforms probabilistic backpropagation, Monte Carlo dropout, deep ensemble, distribution-free ensembles, and hyper ensemble methods on a number of regression benchmarks.

show abstract

Neural Ensemble Search for Uncertainty Estimation and Dataset Shift

Cited by 3 publications

References 0 publications

Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs

Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs

Evolutionary Neural Cascade Search across Supernetworks

AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification

Contact Info

Product

Resources

About