Global Consensus Monte Carlo

Abstract

To conduct Bayesian inference with large datasets, it is often convenient or necessary to distribute the data across multiple machines. We consider a likelihood function expressed as a product of terms, each associated with a subset of the data. Inspired by global variable consensus optimization, we introduce an instrumental hierarchical model associating auxiliary statistical parameters with each term, which are conditionally independent given the top-level parameters. One of these top-level parameters controls the unconditional strength of association between the auxiliary parameters. This model leads to a distributed MCMC algorithm on an extended state space yielding approximations of posterior expectations. A tradeoff between computational tractability and fidelity to the original model can be controlled by changing the association strength in the instrumental model. We further propose the use of an SMC sampler with a sequence of association strengths, allowing both the automatic determination of appropriate strengths and for a bias correction technique to be applied. In contrast to similar distributed Monte Carlo algorithms, this approach requires few distributional assumptions. The performance of the algorithms is illustrated with a number of simulated examples. Supplementary materials for this article are available online.

DG-LMC: A Turn-key and Scalable Synchronous Distributed MCMC Algorithm via Langevin Monte Carlo within Gibbs

“…This step turns to be the milestone of one-shot approaches and was the topic of multiple contributions (Wang and Dunson, 2013;Neiswanger et al, 2014;Minsker et al, 2014;Srivastava et al, 2015;Scott et al, 2016;Nemeth and Sherlock, 2018). Unfortunately, the latter are either infeasible in high-dimensional settings or have been shown to yield inaccurate posterior representations empirically, if the posterior is not near-Gaussian, or if the local posteriors differ significantly (Wang et al, 2015;Dai et al, 2019;Rendell et al, 2021). Alternative schemes have been recently proposed to tackle these issues but their theoretical scaling w.r.t.…”

“…To sample from π given by (1) in a distributed fashion, a large number of approximate methods have been proposed in the past ten years (Neiswanger et al, 2014;Ahn et al, 2014;Rabinovich et al, 2015;Scott et al, 2016;Nemeth and Sherlock, 2018;Chowdhury and Jermaine, 2018;Rendell et al, 2021). Despite multiple research lines, to the best of authors' knowledge, none of these proposals has been proven to be satisfactory.…”

“…However, two main challenges remain: one has to sample efficiently from the conditional distribution Π ρ (z i | θ) for i ∈ [b] and avoid too frequent communication rounds on the master. Existing AXDA-based approaches unfortunately do not fulfill these important requirements Rendell et al, 2021). In this work, we leverage these issues by considering the use of the Langevin Monte Carlo (LMC) algorithm to approximately sample from Π ρ (z i | θ) (Rossky et al, 1978;Roberts and Tweedie, 1996).…”

“…1, this condition unfortunately leads to prohibitive computational costs and hence prevents its practical use for general Bayesian inference problems. Instead of sampling exactly from (4), Rendell et al (2021) rather proposed to use Metropolis-Hastings algorithms. However, it is not clear whether this choice indeed leads to efficient sampling schemes.…”

“…Only using a single step of LMC on each worker might incur important communication costs. To mitigate the latter while increasing the proportion of time spent on exploring the state-space, we instead allow each worker to perform N i ≥ 1 LMC steps (Dieuleveut and Patel, 2019;Rendell et al, 2021). Letting N i varies across workers prevents Algorithm 1 to suffer from a significant block-by-the-slowest delay in cases where the response times of the workers are unbalanced (Ahn et al, 2014).…”