Loop Corrections in Spin Models through Density Consistency

Abstract

Computing marginal distributions of discrete or semi-discrete Markov Random Fields (MRF) is a fundamental, generally intractable, problem with a vast number of applications on virtually all fields of science. We present a new family of computational schemes to calculate approximately marginals of discrete MRFs. This method shares some desirable properties with Belief Propagation, in particular providing exact marginals on acyclic graphs; but at difference with it, it includes some loop corrections, i.e. it takes into account correlations coming from all cycles in the factor graph. It is also similar to Adaptive TAP, but at difference with it, the consistency is not on the first two moments of the distribution but rather on the value of its density on a subset of values. Results on random connectivity and finite dimensional Ising and Edward-Anderson models show a significant improvement with respect to the Bethe-Peierls (tree) approximation in all cases, and with respect to Plaquette Cluster Variational Method approximation in many cases. In particular, for the critical inverse temperature βc of the homogeneous hypercubic lattice, the expansion of (dβc) −1 around d = ∞ of the proposed scheme is exact up to the d −4 order, whereas the two latter are exact only up to the d −2 order.

A Density Consistency approach to the inverse Ising problem

“…O 2 N ): for very small systems the computation can be performed explicitly but for larger (and often more interesting) systems one may rely on a Monte Carlo Markov Chain (MCMC) estimate that, if performed at each update of the parameters, can be time demanding. We propose in the following an analytic expression of the target model parameters such that the maximum likelihood equations in ( 6) is satisfied, when the model statistics is evaluated by the Density Consistency approximation [28].…”

“…Among all the possible choices, in [28] the authors found that any scheme satisfying µi Σii = atanh x i q (ij) gives exact marginalization on acyclic graphs (trees). They then constructed a set of reasonable matching conditions to close the set of update equations (note that in each approximate factor φ ij (x i , x j ) five parameters have to be determined):…”

“…for all distinct pairs i < j. The parameter η ∈ [0, 1] in (17c) acts an interpolation between a full DC solution η = 1 and a BP solution η = 0: indeed, setting η = 0 is equivalent to assume that the cavity distribution is factorized over the edges, and it gives the BP fixed point on any topology [28]. For reasons that will be clear in the following section, we present another update scheme, directly inspired by standard EP implementations, that is obtained by imposing first and second moments matching between q (x) and each q (ij) (x):…”

“…The purpose of this work is to describe another way of solving the IIP using Density Consistency (DC), a novel class of approximation schemes introduced in [28] to compute approximately marginal probability distributions over discrete Markov Random Fields. Its derivation is similar to the well-known Expectation Propagation algorithm [29,30] with the main difference of relying on a modified consistency condition, constructed in such a way that the computation of marginal is exact in the case of acyclic graphs.…”

Compressed sensing reconstruction using Expectation Propagation

“…where in the last step we restricted the structure of the prior P (w) := N i=1 ψ i (w i ) to a factorized form, although more general structures can be considered, e.g. as in [34]. In this work, we have considered the so-called spike-and-slab prior [26]:…”