Parallel Markov chain Monte Carlo for Bayesian hierarchical models with big data, in two stages

Abstract

Due to the escalating growth of big data sets in recent years, new Bayesian Markov chain Monte Carlo (MCMC) parallel computing methods have been developed. These methods partition large data sets by observations into subsets. However, for Bayesian nested hierarchical models, typically only a few parameters are common for the full data set, with most parameters being groupspecific. Thus, parallel Bayesian MCMC methods that take into account the structure of the model and split the full data set by groups rather than by observations are a more natural approach for analysis. Here, we adapt and extend a recently introduced two-stage Bayesian hierarchical modeling approach, and we partition complete data sets by groups. In stage 1, the group-specific parameters are estimated independently in parallel. The stage 1 posteriors are used as proposal distributions in stage 2, where the target distribution is the full model. Using three-level and four-level models, we show in both simulation and real data studies that results of our method agree closely with the full data analysis, with greatly increased MCMC efficiency and greatly reduced computation times. The advantages of our method versus existing parallel MCMC computing methods are also described.

Inverse Estimation Method of Material Randomness Using Observation

“…p(data|θ) is the probability of observing "data" given θ, and is called the likelihood. The likelihood function is a function of the evidence, data, while the posterior probability is a function of the hypothesis, θ and can be expressed by the process as shown in Figure 3 [40].…”

Federated Generalized Bayesian Learning via Distributed Stein Variational Gradient Descent

“…Generalized Bayesian learning aims at minimizing a free energy function over a posterior distribution of the model parameters, including standard Bayesian learning as a special case (Knoblauch et al, 2019). Existing (generalized) Bayesian federated learning protocols are either based on Variational Inference (VI) (Angelino et al, 2016;Neiswanger et al, 2015;Broderick et al, 2013;Corinzia & Buhmann, 2019b) or Monte Carlo (MC) sampling Mesquita et al, 2020;Wei & Conlon, 2019). State-of-the-art methods in either category include Partitioned Variational Inference (PVI), which has been recently introduced as a unifying distributed VI framework that relies on the optimization over parametric posteriors; and Distributed Stochastic Gradient Langevin Dynamics (DSGLD), which is an MC sampling technique that maintains a number of Markov chains updated via local Stochastic Gradient Descent (SGD) with the addition of Gaussian noise Welling & Teh, 2011).…”