Statistical foundation of Variational Bayes neural networks

Bhattacharya, Shrijita; Maiti, Tapabrata

doi:10.1016/j.neunet.2021.01.027

Cited by 9 publications

(3 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The above representation is similar to the proof of Theorems 3.1 and 3.2 in Bhattacharya and Maiti (2021).…”

Section: Proof Of Theorem 44supporting

confidence: 52%

Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details

Jantre¹,

Bhattacharya²,

Maiti³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Sparse deep neural networks have proven to be efficient for predictive model building in large-scale studies. Although several works have studied theoretical and numerical properties of sparse neural architectures, they have primarily focused on the edge selection. Sparsity through edge selection might be intuitively appealing; however, it does not necessarily reduce the structural complexity of a network. Instead pruning excessive nodes in each layer leads to a structurally sparse network which would have lower computational complexity and memory footprint. We propose a Bayesian sparse solution using spike-and-slab Gaussian priors to allow for node selection during training. The use of spike-and-slab prior alleviates the need of an ad-hoc thresholding rule for pruning redundant nodes from a network. In addition, we adopt a variational Bayes approach to circumvent the computational challenges of traditional Markov Chain Monte Carlo (MCMC) implementation. In the context of node selection, we establish the fundamental result of variational posterior consistency together with the characterization of prior parameters. In contrast to the previous works, our theoretical development relaxes the assumptions of the equal number of nodes and uniform bounds on all network weights, thereby accommodating sparse networks with layer-dependent node structures or coefficient bounds. With a layer-wise characterization of prior inclusion probabilities, we also discuss optimal contraction rates of the variational posterior. Finally, we provide empirical evidence to substantiate that our theoretical work facilitates layer-wise optimal node recovery together with competitive predictive performance.

show abstract

“…The above representation is similar to the proof of Theorems 3.1 and 3.2 in Bhattacharya and Maiti (2021).…”

Section: Proof Of Theorem 44supporting

confidence: 52%

Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details

Jantre¹,

Bhattacharya²,

Maiti³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Theoretical results around variational inference have mostly centered around its statistical properties, including asymptotic properties (Alquier and Ridgway, 2017;Alquier et al, 2016;Banerjee et al, 2021;Bhattacharya et al, 2020;Bhattacharya and Maiti, 2021;Bickel et al, 2013;Campbell and Li, 2019;Celisse et al, 2012;Chen and Ryzhov, 2020;Chérief-Abdellatif, 2019, 2020Chérief-Abdellatif et al, 2018;Guha et al, 2020;Hajargasht, 2019;Hall et al, 2011a,b;Han and Yang, 2019;Jaiswal et al, 2020;Knoblauch, 2019;Pati et al, 2017;Wang and Titterington, 2004;Wang et al, 2006;Blei, 2019, 2018;Westling and McCormick, 2015;Womack et al, 2013;Yang et al, 2017;You et al, 2014;Zhang and Gao, 2017), nite sample approximation error (Chen et al, 2017;Giordano et al, 2017;Huggins et al, 2020Huggins et al, , 2018Sheth and Khardon, 2017), robustness to model misspeci cation (Alquier and Ridgway, 2017;Chérief-Abdellatif et al, 2018;Medina et al, 2021;Wang and Blei, 2019), and properties in high-dimensional settings (Mukherjee and Sen, 2021;Ray and Szabó, 2021;Ray et al, 2020;…”

Section: Main Ideasmentioning

confidence: 99%

Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection

Bhatia¹,

Kuang²,

Ma³

et al. 2022

Preprint

View full text Add to dashboard Cite

Variational inference has recently emerged as a popular alternative to the classical Markov chain Monte Carlo (MCMC) in large-scale Bayesian inference. The core idea of variational inference is to trade statistical accuracy for computational e ciency. It aims to approximate the posterior, reducing computation costs but potentially compromising its statistical accuracy. In this work, we study this statistical and computational trade-o in variational inference via a case study in inferential model selection. Focusing on Gaussian inferential models (also known as variational approximating families) with diagonal plus low-rank precision matrices, we initiate a theoretical study of the trade-o s in two aspects, Bayesian posterior inference error and frequentist uncertainty quanti cation error. From the Bayesian posterior inference perspective, we characterize the error of the variational posterior relative to the exact posterior. We prove that, given a xed computation budget, a lower-rank inferential model produces variational posteriors with a higher statistical approximation error, but a lower computational error; it reduces variances in stochastic optimization and, in turn, accelerates convergence. From the frequentist uncertainty quanti cation perspective, we consider the precision matrix of the variational posterior as an uncertainty estimate. We nd that, relative to the true asymptotic precision, the variational approximation su ers from an additional statistical error originating from the sampling uncertainty of the data. Moreover, this statistical error becomes the dominant factor as the computation budget increases. As a consequence, for small datasets, the inferential model need not be full-rank to achieve optimal estimation error (even with unlimited computation budget). We nally demonstrate these statistical and computational trade-o s inference across empirical studies, corroborating the theoretical ndings.

show abstract

“…distributions for positive variables. Despite its simplicity, mean-field variational family can approximate a wide class of posteriors and is good enough to achieve consistency for the variational posterior for a wide class of models (Wang and Blei, 2018;Zhang and Gao, 2020;Bhattacharya and Maiti, 2021). Moreover, all the strictly positive variational parameters λ will be transformed as λ = log(e λ − 1) (13) to avoid constrained optimization.…”

Section: Implementation Details and Variational Familiesmentioning

confidence: 99%

Black Box Variational Bayesian Model Averaging

Kejzlar¹,

Bhattacharya²,

Son³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

For many decades now, Bayesian Model Averaging (BMA) has been a popular framework to systematically account for model uncertainty that arises in situations when multiple competing models are available to describe the same or similar physical process. The implementation of this framework, however, comes with multitude of practical challenges including posterior approximation via Markov Chain Monte Carlo and numerical integration. We present a Variational Bayes Inference approach to BMA as a viable alternative to the standard solutions which avoids many of the aforementioned pitfalls. The proposed method is "black box" in the sense that it can be readily applied to many models with little to no model-specific derivation. We illustrate the utility of our variational approach on a suite of standard examples and discuss all the necessary implementation details. Fully documented Python code with all the examples is provided as well.

show abstract

Statistical foundation of Variational Bayes neural networks

Cited by 9 publications

References 16 publications

Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details

Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details

Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection

Black Box Variational Bayesian Model Averaging

Contact Info

Product

Resources

About