Gaussian variational approximation with sparse precision matrices

Abstract

We consider the problem of learning a Gaussian variational approximation to the posterior distribution for a high-dimensional parameter, where we impose sparsity in the precision matrix to reflect appropriate conditional independence structure in the model. Incorporating sparsity in the precision matrix allows the Gaussian variational distribution to be both flexible and parsimonious, and the sparsity is achieved through parameterization in terms of the Cholesky factor. Efficient stochastic gradient methods which make appropriate use of gradient information for the target distribution are developed for the optimization. We consider alternative estimators of the stochastic gradients which have lower variation and are more stable. Our approach is illustrated using generalized linear mixed models and state space models for time series.Comment: 18 pages, 9 figure

Bayesian Variational Inference for Exponential Random Graph Models

“…Unbiased gradient estimates can be constructed in several ways depending on whether Roeder et al (2017), Tan and Nott (2018) and Tan (2018), an advantage of estimating both terms in (8) using the same samples s ∼ φ(s) is that the stochasticity arising from s in the two terms cancel out each other resulting in smaller variation in the gradients at convergence. As log q λ (θ) depends on {µ, C} directly as well as through θ, we apply chain rule to obtain…”

Gaussian Variational Approximation With a Factor Covariance Structure

“…a diagonal approximation) are very different, and even a crude allowance for posterior correlation with a small number of factors can grealy improve estimation of the posterior marginal distributions. Finally, we also compare the variational marginal density of the regression coefficients with the method in Tan and Nott (2016). The method of Tan and Nott (2016) gives similar answers to MCMC in this example, as shown in Figure 5 of their manuscript, so the Tan and Nott (2016) can be considered both a gold standard for a normal approximation as well as a good gold standard more globally.…”

“…which shows that a gradient estimate based on a single sample of f using (9) and (10) will be zero at the mode. Similar points are discussed in Salimans and Knowles (2013), Han et al (2016) and Tan and Nott (2016) in other contexts and we use the gradient estimates based on (9) and (10) (6), (9) and (10) at λ (t) where the expectations are approximated from the single sample ( (t) , z (t) ).…”

“…edu/statdata/statdata/stat-_logistic.html, and contains data on 500 subjects, who were followed over seven years. Following Tan and Nott (2016), we consider a logistic mixed effects model of the form where p(β) is N (0, 100I 8 ), p(ζ) is N (0, 100) and p(u i |ζ) is N (0, exp(2ζ)).…”

“…The blocks of the mean vector and non-zero blocks of the precision matrix are parametrized in terms of global parameters that relate them to local data -an example of so-called amortized variational inference -which was also introduced in Kingma and Welling (2014). Tan and Nott (2016) parametrize the variational optimization directly in terms of the Cholesky factor of the precision matrix and impose sparsity on the Cholesky factor that reflects conditional independence relationships. They show how the sparsity can be exploited in the computation of gradients with the reparametrization trick.…”