Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

Zhang, Ruqi; Li, Chunyuan; Zhang, Jianyi; Chen, Changyou; Wilson, Andrew Gordon

doi:10.48550/arxiv.1902.03932

Cited by 39 publications

(59 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For this case of model misspecification, a cold posterior performs better in terms of predictive capacity (MPL). Cold posteriors have been found to perform well also in other studies [146,148,149]. In Fig.…”

Section: Metricsupporting

confidence: 74%

“…In this regard, a technique that is often used in practice is posterior tempering, i.e., sampling θ values from p(θ|D) 1/τ instead of the true posterior (τ = 1), where τ is called temperature. Specifically, it has been reported in the literature that "cold" posteriors, τ < 1, perform better [146,148,149], although using a more informed prior can potentially remove this effect [143]. Cold posteriors can be interpreted as over-counting the available data using 1/τ replications of it, thus, making the posterior more concentrated.…”

Section: A4 Posterior Tempering For Model Misspecificationmentioning

confidence: 99%

“…Concurrently with the present work, the replica-exchange MCMC technique has been employed in conjunction with DeepONet for operator learning [92]. Additional UQ methods based on MCMC can be found, indicatively, in [35,149,[163][164][165].…”

Section: B21 Markov Chain Monte Carlo (Mcmc)mentioning

confidence: 99%

See 2 more Smart Citations

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. To demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed, which we will make available as open-source library of all codes included in this framework.

show abstract

Section: Metricsupporting

confidence: 74%

Section: A4 Posterior Tempering For Model Misspecificationmentioning

confidence: 99%

See 1 more Smart Citation

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In particular, we take the code 7 and networks from Fortuin et al (2021b,a) and mirror their experimental setup as closely as possible. This code combines a cyclical learning rate schedule (Zhang et al, 2019), a gradient-guided Monte Carlo (GG-MC) scheme (Garriga-Alonso & Fortuin, 2021), and the preconditioning and convergence diagnostics from Wenzel et al (2020). Following Fortuin et al (2021b), we ran 60 cycles with 45 epochs in each cycle.…”

Section: Bayesian Neural Network and The Cold Posterior Effectmentioning

confidence: 99%

Data augmentation in Bayesian neural networks and the cold posterior effect

Nabarro¹,

Stoil²,

Garriga-Alonso³

et al. 2021

Preprint

View full text Add to dashboard Cite

Data augmentation is a highly effective approach for improving performance in deep neural networks. The standard view is that it creates an enlarged dataset by adding synthetic data, which raises a problem when combining it with Bayesian inference: how much data are we really conditioning on? This question is particularly relevant to recent observations linking data augmentation to the cold posterior effect. We investigate various principled ways of finding a log-likelihood for augmented datasets. Our approach prescribes augmenting the same underlying image multiple times, both at test and train-time, and averaging either the logits or the predictive probabilities. Empirically, we observe the best performance with averaging probabilities. While there are interactions with the cold posterior effect, neither averaging logits or averaging probabilities eliminates it. IntroductionData augmentation (Shorten & Khoshgoftaar, 2019) is a fundamental technique for obtaining high performance in modern neural networks (NNs). In computer vision, data augmentation involves creating synthetic training examples by making small modifications, such as a rotation or crop, to the input image.At the same time, Bayesian inference allows us to reason about uncertainty in neural network weights (MacKay, 1992;Welling & Teh, 2011;Blundell et al., 2015;Fortuin, 2021) given limited data. Bayesian inference is particularly important in safety-critical settings such as self-driving cars or medical imaging where it is crucial to be able to hand over to a human when uncertainty is too large. * equal contribution † equal contributionPreprint. Under review.

show abstract

“…Likewise, the Laplace inference for BBNs has improved in scalability using further GGN approximations [30,31,37,51,52] and sub-network inference [12,37]. Orthogonally, MCMC methods for BNNs have been improved [20,24,61,66], better BNN priors have been studied [19,21], and even deep ensembles [39] have been cast as approximate inference [9-11, 32, 49, 50, 62].…”

Section: Related Workmentioning

confidence: 99%

Pathologies in priors and inference for Bayesian transformers

Cinquin¹,

Immer²,

Horn³

et al. 2021

Preprint

View full text Add to dashboard Cite

In recent years, the transformer has established itself as a workhorse in many applications ranging from natural language processing to reinforcement learning. Similarly, Bayesian deep learning has become the gold-standard for uncertainty estimation in safety-critical applications, where robustness and calibration are crucial. Surprisingly, no successful attempts to improve transformer models in terms of predictive uncertainty using Bayesian inference exist. In this work, we study this curiously underpopulated area of Bayesian transformers. We find that weight-space inference in transformers does not work well, regardless of the approximate posterior. We also find that the prior is at least partially at fault, but that it is very hard to find well-specified weight priors for these models. We hypothesize that these problems stem from the complexity of obtaining a meaningful mapping from weight-space to function-space distributions in the transformer. Therefore, moving closer to function-space, we propose a novel method based on the implicit reparameterization of the Dirichlet distribution to apply variational inference directly to the attention weights. We find that this proposed method performs competitively with our baselines.

show abstract

Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

Cited by 39 publications

References 25 publications

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Data augmentation in Bayesian neural networks and the cold posterior effect

Pathologies in priors and inference for Bayesian transformers

Contact Info

Product

Resources

About