Bayesian Deep Net GLM and GLMM

Abstract

Deep feedforward neural networks (DFNNs) are a powerful tool for functional approximation. We describe flexible versions of generalized linear and generalized linear mixed models incorporating basis functions formed by a DFNN. The consideration of neural networks with random effects is not widely used in the literature, perhaps because of the computational challenges of incorporating subject specific parameters into already complex models. Efficient computational methods for high-dimensional Bayesian inference are developed using Gaussian variational approximation, with a parsimonious but flexible factor parametrization of the covariance matrix. We implement natural gradient methods for the optimization, exploiting the factor structure of the variational covariance matrix in computation of the natural gradient. Our flexible DFNN models and Bayesian inference approach lead to a regression and classification method that has a high prediction accuracy, and is able to quantify the prediction uncertainty in a principled and convenient way. We also describe how to perform variable selection in our deep learning method. The proposed methods are illustrated in a wide range of simulated and real-data examples, and the results compare favourably to a state of the art flexible regression and classification method in the statistical literature, the Bayesian additive regression trees (BART) method. User-friendly software packages in Matlab, R and Python implementing the proposed methods are available at https://github.com/VBayesLab.

Horseshoe Regularisation for Machine Learning in Complex and Deep Models1

“…Deep models have also been proposed for glms, which is a flexible technique for modeling data distributed according to exponential family distributions, and can model a large class of response variables, including real-valued, categorical or counts. Tran et al (2018) develop flexible versions for glms using the output of a feedforward neural network. With responses y and predictors X, a conventional glm models E(y | X) = g(Xβ), where β are the regression coefficients and g(·) is the link function.…”

“…Thus, the conditional mean of the responses is a linear function of X, transformed through the link function g. This linearity assumption is often restrictive and a natural way to introduce nonlinearity is by replacing X with the output of a multi-layer feedforward neural network that has X as input and consequently, whose output is a nonlinear function of X. Tran et al (2018) term this model DeepGLM. Similar to deep neural networks, a variational approximation to the log likelihood is used for training the model and global-local priors are used for inducing sparsity on β.…”

“…Ingraham and Marks (2017) developed the "fadeout" procedure for variational inference under horseshoe priors in undirected graphical models, completely circumventing the need for MCMC. The deep-GLM model of Tran et al (2018) also applies a variational inference procedure for regularizing the weights in a deep neural network whose output is then fed to a glm.…”

Comparison of Deep Neural Networks and Deep Hierarchical Models for Spatio-Temporal Data

“…Chatzis, 2015;Chien and Ku, 2016;Gan et al, 2016;McDermott and Wikle, 2017a) but are quite sensitive to particular data sets and are typically computationally prohibitive. More recently, approximate Bayesian methods such as variational Bayes (Tran et al, 2018), and scalable Bayesian methods (Snoek et al, 2015) have been used successfully in deep models. In the context of DN-DSTMs this is still an active area of research.…”

Deep integro-difference equation models for spatio-temporal forecasting

“…The similarities and differences discussed above have helped establish a new branch in Statistical Science that looks at combining formal statistical models with the flexibility of deep ML models with a view to exploiting the strengths of both approaches in a single framework for better prediction and forecasting. For example, Nguyen et al (2019) use a type of RNN known as the long short-term memory (LSTM) within a classic stochastic volatility model in order to cater for long-range (temporal) dependence, while Tran et al (2019) (2019) used a deep structure to model nonstationarity in spatial models. The resulting model is interpretable, but the framework is firmly seated within the standard geostatistical setting where time, if considered, would be treated as an extra dimension; such models tend to be ill-suited for forecasting purposes.…”