Localization for MCMC: sampling high-dimensional posterior distributions with local structure

Abstract

We investigate how ideas from covariance localization in numerical weather prediction can be used in Markov chain Monte Carlo (MCMC) sampling of high-dimensional posterior distributions arising in Bayesian inverse problems. To localize an inverse problem is to enforce an anticipated "local" structure by (i ) neglecting small off-diagonal elements of the prior precision and covariance matrices; and (ii ) restricting the influence of observations to their neighborhood. For linear problems we can specify the conditions under which posterior moments of the localized problem are close to those of the original problem. We explain physical interpretations of our assumptions about local structure and discuss the notion of high dimensionality in local problems, which is different from the usual notion of high dimensionality in function space MCMC. The Gibbs sampler is a natural choice of MCMC algorithm for localized inverse problems and we demonstrate that its convergence rate is independent of dimension for localized linear problems. Nonlinear problems can also be tackled efficiently by localization and, as a simple illustration of these ideas, we present a localized Metropolis-within-Gibbs sampler. Several linear and nonlinear numerical examples illustrate localization in the context of MCMC samplers for inverse problems.

Accelerating Metropolis-within-Gibbs sampler with localized computations of differential equations

“…In this paper, we discuss how to infer initial conditions of high dimensional SDEs with local interactions, when noisy observations are available. Previous research [18] has shown that Metropoliswithin-Gibbs (MwG) sampling has dimension independent convergence rate. However, each MwG iteration requires a computation cost of O(n 2 ), with n being the model dimension.…”

“…In [18], it is shown that MwG has dimension independent performance if 1) p 0 is a Gaussian distribution, and its covariance or precision matrix is close to be banded; 2) each component of y has significant dependence only on a few components of x. It is also discussed how to truncate the far-off diagonal entries of the prior covariance or precision matrix.…”

“…Because (18) only requires execution for j ∈ B i⋆ , so the total cost of obtainingx l is only O(1), which is one order cheaper than obtainingx p directly with Euler-Maruyama. This partial computation naturally introduces an error.…”

“…The solution of Lorenz 96 model has an equilibrium distribution, which can be obtained by longtime simulation. We use a Gaussian approximation of it as the prior distribution, of which the mean vector and covariance matrix are obtained by the localization procedure described in [18]. We consider solving the inverse problem formulated as (5), where the underlying model (4) is given by (24) with T = 0.4.…”

Bayesian Update with Importance Sampling: Required Sample Size

“…Local particle filters are reviewed in [ 21 ] and their potential to beat the curse of dimension is investigated from a theoretical viewpoint in [ 16 ]. Localization is popular in ensemble Kalman filters [ 20 ] and has been employed in Markov chain Monte Carlo [ 22 , 23 ]. Our focus in this paper is not on localization but rather on providing a unified and accessible understanding of the roles that dimension, noise-level and other model parameters play in approximating the Bayesian update.…”