Variational Inference for Generalized Linear Mixed Models Using Partially Noncentered Parametrizations

Abstract

The effects of different parametrizations on the convergence of Bayesian computational algorithms for hierarchical models are well explored. Techniques such as centering, noncentering and partial noncentering can be used to accelerate convergence in MCMC and EM algorithms but are still not well studied for variational Bayes (VB) methods. As a fast deterministic approach to posterior approximation, VB is attracting increasing interest due to its suitability for large high-dimensional data. Use of different parametrizations for VB has not only computational but also statistical implications, as different parametrizations are associated with different factorized posterior approximations. We examine the use of partially noncentered parametrizations in VB for generalized linear mixed models (GLMMs). Our paper makes four contributions. First, we show how to implement an algorithm called nonconjugate variational message passing for GLMMs. Second, we show that the partially noncentered parametrization can adapt to the quantity of information in the data and determine a parametrization close to optimal. Third, we show that partial noncentering can accelerate convergence and produce more accurate posterior approximations than centering or noncentering. Finally, we demonstrate how the variational lower bound, produced as part of the computation, can be useful for model selection.

Variational methods for fitting complex Bayesian mixed effects models to health data

“…The MFVB methods we considered here are fast and versatile and can be easily extended to more complicated scenarios. For example, the methods allow arbitrary priors for the hyperparameters [57] and similar types of model with Gaussian responses [18,20,21]. Stewart [22] provides great examples of more elaborate models within the context of social sciences.…”

“…Armagan and Dunson [19] developed a fast remedy for sparse covariance estimation relying on a decomposition but is restricted to linear response. Tan and Nott [20] extended their approach and introduced a partially noncentered nonparametrization strategy for generalized linear mixed models, allowing the random effects for each group to be independent. Such restriction was shown to improve efficiency of the variational algorithms.…”

Parallel Variational Bayes for Large Datasets With an Application to Generalized Linear Mixed Models

“…GLMMs are often considered difficult to estimate because of the presence of random effects and lack of conjugate priors. VB schemes for GLMMs were considered previously by Rijmen and Vomlel (2008); Ormerod and Wand (2012) and Tan and Nott (2013b), and shown to have attractive computational and accuracy trade-offs.…”

Laplace Variational Approximation for Semiparametric Regression in the Presence of Heteroscedastic Errors

“…Examples of such models include logistic regression (Jaakkola and Jordan, 1997), nonparametric regression with measurement error (Pham et al, 2013), generalized linear mixed models (Tan and Nott, 2013), and correlated topic models (Blei and Lafferty, 2007). Variational approximations of non-conjugate models often have to be derived on a case by case basis and can be difficult to handle.…”