Annealed Leap-Point Sampler for Multimodal Target Distributions

Abstract

In Bayesian statistics, exploring multimodal posterior distribution poses major challenges for existing techniques such as Markov Chain Monte Carlo (MCMC). These problems are exacerbated in high-dimensional settings where MCMC methods typically rely upon localised proposal mechanisms. This paper introduces the Annealed Leap-Point Sampler (ALPS), which augments the target distribution state space with modified annealed (cooled) target distributions, in contrast to traditional approaches which have employed tempering. The temperature of the coldest state is chosen such that its corresponding annealed target density can be sufficiently well-approximated by a Laplace approximation. As a result, a Gaussian mixture independence Metropolis-Hastings sampler can perform mode-jumping proposals even in high-dimensional problems. The ability of this procedure to mode hop at this super-cold state is then filtered through to the target state using a sequence of tempered targets in a similar way to that in parallel tempering methods. ALPS also incorporates the best aspects of current gold-standard approaches to multimodal sampling in high-dimensional contexts. A theoretical analysis of the ALPS approach in high dimensions is given, providing practitioners with a gauge on the optimal setup as well as the scalability of the algorithm. For a d-dimensional problem the it is shown that the coldest inverse temperature level required for the ALPS only needs to be linear in the dimension, O(d), and this means that for a collection of multimodal problems the algorithmic cost is polynomial, O d 3 . ALPS is illustrated on a complex multimodal posterior distribution that arises from a seemingly-unrelated regression (SUR) model of longitudinal data from U.S. manufacturing firms.

Skew brownian motion and complexity of the alps algorithm

“…Remark 1. The 'vanilla' ALPS algorithm studied herein differs in certain ways from the full ALPS algorithm for actual applications in [29]. For example, we assume the process mixes perfectly between modes when β = β (d) max (due to near-Gaussianity and the algorithm's auxiliary mode-jumping steps), and not at all when β < β (d) max , while the full algorithm mixes better and better at higher β values but never perfectly.…”

“…[2, Section 4]). Finally, the full ALPS algorithm in [29] also makes use of the QuanTA transformation [30], an additional affine transformation to increase the efficiency of the temperature-swap moves, which we omit here; we discuss the effect of this extra QuanTA transformation in Corollary 3.…”

“…We consider a version of the ALPS algorithm of [29]. We assume the chain always mixes immediately within each mode, but the chain can only jump between modes when at the sufficiently cold inverse temperature β = β (d) max , at which point it immediately jumps to any of its modes with the correct probability weight.…”

“…Tempering methods for MCMC usually consider powers π β for small values β ∈ (0, 1], to make the target distribution flatter and thus allow for easier mixing between modes. By contrast, [29] introduces the ALPS algorithm, which instead uses large values β 1, combined with locally weight-preserving tempering distributions as in [31] so the modes retain their relative masses. These choices make the modes of π even more separated.…”

“…An alternative approach, the Annealed Leap-Point Sampler (ALPS), is introduced in [29]. This algorithm instead considers very large values β 1, corresponding to very peaked target densities at very cold temperatures.…”