What Are Bayesian Neural Network Posteriors Really Like?

Abstract

The posterior over Bayesian neural network (BNN) parameters is extremely high-dimensional and non-convex. For computational reasons, researchers approximate this posterior using inexpensive mini-batch methods such as meanfield variational inference or stochastic-gradient Markov chain Monte Carlo (SGMCMC). To investigate foundational questions in Bayesian deep learning, we instead use full-batch Hamiltonian Monte Carlo (HMC) on modern architectures. We show that (1) BNNs can achieve significant performance gains over standard training and deep ensembles; (2) a single long HMC chain can provide a comparable representation of the posterior to multiple shorter chains; (3) in contrast to recent studies, we find posterior tempering is not needed for near-optimal performance, with little evidence for a "cold posterior" effect, which we show is largely an artifact of data augmentation; (4) BMA performance is robust to the choice of prior scale, and relatively similar for diagonal Gaussian, mixture of Gaussian, and logistic priors; (5) Bayesian neural networks show surprisingly poor generalization under domain shift; (6) while cheaper alternatives such as deep ensembles and SGMCMC can provide good generalization, they provide distinct predictive distributions from HMC. Notably, deep ensemble predictive distributions are similarly close to HMC as standard SGLD, and closer than standard variational inference.

Bayesian Learning via Neural Schrödinger-Föllmer Flows

“…We ran experiments on Bayesian NN regression, classification, logistic regression and ICA (Amari et al, 1996), reporting accuracies, log joints (Welling and Teh, 2011;Izmailov et al, 2021) and expected calibration error (ECE) (Guo et al, 2017). For details on exact experimental setups please see Appendix F. Across experiments we compare to SGLD as (Izmailov et al, 2021). In the Bayesian NN tasks the likelihood is parametrised via p(y…”

Online, Informative MCMC Thinning with Kernelized Stein Discrepancy

“…We measure the quality of each sampling method's approximation to the predictive distribution corresponding to the true posterior. We generate the ground truth predictive distribution by running 20,000 samples of SGLD (Welling & Teh, 2011), and follow (Izmailov et al, 2021) by measuring the top-1 agreement and total variation with respect to the ground truth predictive distribution. Total variation is…”

“…The task is 10-class image classification, and we use a ResNet-20 with Filter Response Normalization from (Izmailov et al, 2021) as our base model. Our results in Figure 5 demonstrate that our online thinning method outperforms both the baseline sampler and SPMCMC-based samplers on the agreement metric.…”

“…The task is two-class sentiment prediction and we use a CNN-LSTM from (Izmailov et al, 2021) as our base model. Our results in Figure 6 demonstrate that all of our online thinning methods, in particular KSDT-SQRT, improve the Pareto frontier of the corresponding baseline sampler on both metrics.…”

Stochastic Neural Networks with Infinite Width are Deterministic

“…This implies that good Bayesian inference can only be made if the statistics of y are correctly modeled. For example, stochastic neural networks are expected to have well-calibrated uncertainty estimates, a trait that is highly desirable for practical safe and reliable applications (Wilson & Izmailov, 2020;Gawlikowski et al, 2021;Izmailov et al, 2021). This expectation means that a well-trained stochastic network should have a predictive variance that matches the actual level of randomness in the labeling.…”