Variational Approximations for Generalized Linear Latent Variable Models

Abstract

Generalized Linear Latent Variable Models (GLLVMs) are a powerful class of models for understanding the relationships among multiple, correlated responses. Estimation however presents a major challenge, as the marginal likelihood does not possess a closed form for non-normal responses. We propose a variational approximation (VA) method for estimating GLLVMs. For the common cases of binary, ordinal, and overdispersed count data, we derive fully closed form approximations to the marginal log-likelihood function in each case. Compared to other methods such as the expectation-maximization algorithm, estimation using VA is fast and straightforward to implement. Predictions of the latent variables and associated uncertainty estimates are also obtained as part of the estimation process. Simulations show that VA estimation performs similar to or better than some currently available methods, both at ACCEPTED MANUSCRIPT predicting the latent variables and estimating their corresponding coefficients. They also show that VA estimation offers dramatic reductions in computation time particularly if the number of correlated responses is large relative to the number of observational units. We apply the variational approach to two datasets, estimating GLLVMs to understanding the patterns of variation in youth gratitude and for constructing ordination plots in bird abundance data. R code for performing VA estimation of GLLVMs is available online.

Order Selection and Sparsity in Latent Variable Models via the Ordered Factor LASSO

“…Additionally, the computational methods proposed in this article may not be feasible in such settings, and instead we may need to combine OFAL with faster, approximate likelihood‐based estimation methods (Hui et al., ; Niku et al., ). Such research into more computationally efficient approaches will also be beneficial if we wanted to include more tuning parameters in the OFAL penalty, for example, the mixing parameter discussed in Section .…”

Semiparametric Regression Using Variational Approximations

“…Ormerod and Wand (2012) considered a logit link and accepted the fact that this resulted in a term, b(ω) = ln{1 + exp(η)}, whose expectation with respect to the normal variational distribution h(β|a, A) would require (univariate) Monte Carlo integration. In an attempt to get around this problem, Hui et al (2017b) considered the probit link and exploited the fact that the model could be reparametrized by introducing a normally distributed auxiliary variable. However, a downside with this approach, which was not picked up by Hui et al (2017b), is that the estimate of A can be quite biased and variability of the posterior distribution of β (as approximated by the variational distribution) tends to be underestimated.…”

“…That is, we use VA to refer to replacing the intractable marginal log-likelihood by a tractable lower bound approximation, which is then treated as the new objective function. The VA approach has only recently been proposed for overcoming intractable marginal likelihood functions, e.g., for generalized linear mixed models (Ormerod and Wand, 2012) and generalized linear latent variable models (Hui et al, 2017b). However to our knowledge, this article is the first to apply VA to semiparametric regression models, let alone study their theoretical properties and finite sample performance.…”

Evaluating and presenting uncertainty in model‐based unconstrained ordination

“…In Hui et al (), four benefits of model‐based approaches to unconstrained ordination are detailed as follows: controlling spurious data properties, model checking, model selection and inference, and efficiency; however, little attention is given to assessing uncertainty in model parameters. In fact, the only mention of confidence intervals in this section states that “accuracy of such confidence intervals in this context is in need of evaluation.” A later paper with a Bayesian implementation (Hui, ) largely resolves the issue of accuracy of the intervals and while other recent articles (Hui, Tanaka, & Warton, ; Hui, Warton, Ormerod, Haapaniemi, & Taskinen, ; Niku, Warton, Hui, & Taskinen, ) do touch on variability, understanding and assessing uncertainty in the latent factors is still not a point of emphasis. Walker () does include an analysis of uncertainty in indirect gradient analysis, but the scope is limited to a single dimensional projection of presence/absence data.…”