Gaussian Variational Approximation With a Factor Covariance Structure

Abstract

Variational approximation methods have proven to be useful for scaling Bayesian computations to large data sets and highly parametrized models. Applying variational methods involves solving an optimization problem, and recent research in this area has focused on stochastic gradient ascent methods as a general approach to implementation.Here variational approximation is considered for a posterior distribution in high dimensions using a Gaussian approximating family. Gaussian variational approximation with an unrestricted covariance matrix can be computationally burdensome in many problems because the number of elements in the covariance matrix increases quadratically with the dimension of the model parameter. To circumvent this problem, low-dimensional factor covariance structures are considered. General stochastic gradient approaches to efficiently perform the optimization are described, with gradient estimates obtained using the so-called "reparametrization trick". The end result is a flexible and efficient approach to high-dimensional Gaussian variational approximation, which we illustrate using eight real datasets.

Bayesian Deep Net GLM and GLMM

“…This means that for the Leukaemia and Breast datasets the dimension of θ is 14,242 so these are examples with a high dimensional parameter. These data were also considered in Ong et al (2017a) where slow convergence in the variational optimization was observed using their method; we show here that a natural gradient approach offers a significant improvement. Ong et al (2017a) and run VAFC with f =4 factors and use only a single sample to estimate the gradient of lower bound (S = 1) in two methods.…”

“…Here the variational optimization is challenging because of the strong dependence between local variance parameters and the corresponding coefficients. Using three real datasets we show that the natural gradient estimation method improves the performance of the approach described in Ong et al (2017a). Let y i ∈{0,1} be a binary response with the corresponding covariates x i =(x i1 ,...,x ip ) , i = 1,...,n. We consider the logistic regression model…”

“…The parameters λ j , j = 1,...,p, are local variance parameters providing shrinkage for individual coefficients, and the parameter g is a global shrinkage parameter which can adapt to the overall level of sparsity. The above prior settings are the same as those considered in Ong et al (2017a). The parameter θ consists of θ=(β 0 ,β ,δ ,γ) , where δ=(δ 1 ,...,δ p ) =(logλ 1 ,...,logλ p ) , and γ = logg.…”

Bayesian Inference for Regression Copulas

“…Successful application of variational methods requires the VA to be computationally and analytically tractable, and an appropriate transformation needs to exist for the re-parameterization trick to be used. Following Ong et al (2018), the Gaussian approximation q λ (ϑ) = φ(ϑ; µ, Υ) with a parsimonious factor covariance structure meets both conditions. Here, Υ = ΨΨ + ∆ 2 , where Ψ is a full rank p ϑ × K matrix with K p ϑ , d = (d 1 , .…”

“…For uniqueness, it is common to also assume Ψ i,i = 1, although we do not because the lack of uniqueness does not hinder the optimization, and the unconstrained parametrization is more convenient. To apply the re-parameterization Ong et al (2018) show that:…”