Split-and-Augmented Gibbs Sampler—Application to Large-Scale Inference Problems

Abstract

This paper derives two new optimization-driven Monte Carlo algorithms inspired from variable splitting and data augmentation. In particular, the formulation of one of the proposed approaches is closely related to the alternating direction method of multipliers (ADMM) main steps. The proposed framework enables to derive faster and more efficient sampling schemes than the current state-of-theart methods and can embed the latter. By sampling efficiently the parameter to infer as well as the hyperparameters of the problem, the generated samples can be used to approximate Bayesian estimators of the parameters to infer. Additionally, the proposed approach brings confidence intervals at a low cost contrary to optimization methods. Simulations on two often-studied signal processing problems illustrate the performance of the two proposed samplers. All results are compared to those obtained by recent state-of-the-art optimization and MCMC algorithms used to solve these problems. Index TermsBayesian inference, data augmentation, high-dimensional problems, Markov chain Monte Carlo, variable splitting.

Efficient Sampling through Variable Splitting-inspired Bayesian Hierarchical Models

“…Thereby, the marginal distribution of x under πρ stands for an approximation of π. The corresponding approximation error, controlled by ρ, can be made arbitrarily small as depicted by Theorem 1 proven in [20].…”

“…Strong similarities with distributed optimization algorithms, e.g. the ADMM, are discussed in [20]. The Gibbs sampler derived to sample from the split distribution πρ is depicted in algo.…”

“…convexity, differentiability) of the potential functions f1 and f2, the coupling function can be adaptively chosen. For instance, [20,22] consider ϕρ(x) = (2ρ 2 ) −1 ∥x − z∥ 2 2 for its gradient Lipschitz, differentiability and convexity properties while [21] invokes conjugacy arguments to choose ϕρ. For a detailed discussion on the Gibbs sampling procedure, we refer the reader to [20, Section III.A].…”

Sparse Bayesian Binary Logistic Regression Using the Split-and-Augmented Gibbs Sampler

“…As stated in [17], SPA can be viewed as a "divide-and-conquer" approach where the initial sampling difficulty is divided in different easier sampling steps. Thus, the conditional distributions associated to β, u1 and u2 are Gaussian with diagonal covariance matrices for the last two distributions.…”

“…These sampling methods are special instances of Metropolis-Hastings algorithms based on the Langevin diffusion process which were improved in [16]. More recently, the connection between simulation-based algorithms and optimization has been strengthened by the so-called split-and-augmented Gibbs sampler (SPA) [17]. This algorithm stands for a general tool to conduct Bayesian inference that uses a "divide-and-conquer" strategy.…”