The Dynamics of Message Passing on Dense Graphs, with Applications to Compressed Sensing

Abstract

Approximate message passing' algorithms have proved to be effective in reconstructing sparse signals from a small number of incoherent linear measurements. Extensive numerical experiments further showed that their dynamics is accurately tracked by a simple one-dimensional iteration termed state evolution. In this paper we provide rigorous foundation to state evolution. We prove that indeed it holds asymptotically in the large system limit for sensing matrices with independent and identically distributed gaussian entries.While our focus is on message passing algorithms for compressed sensing, the analysis extends beyond this setting, to a general class of algorithms on dense graphs. In this context, state evolution plays the role that density evolution has for sparse graphs.The proof technique is fundamentally different from the standard approach to density evolution, in that it copes with a large number of short cycles in the underlying factor graph. It relies instead on a conditioning technique recently developed by Erwin Bolthausen in the context of spin glass theory.for an appropriate sequence of non-linear functions {η t } t≥0 . (Here by convention any variable with negative index is assumed to be 0.) The algorithm succeeds if x t converges to a good approximation of x 0 (cf.[DMM09] for details). Throughout this paper, the matrix A is normalized in such a way that its columns have ℓ 2 norm 1 concentrated around 1. Given a vector x ∈ R N and a scalar function f : R → R, we write f (x) for

Rigorous dynamics of expectation-propagation-based signal recovery from unitarily invariant measurements

“…The proof strategy is based on a conditioning technique used in [12]. A challenging part in the proof is to evaluate the distribution of an estimation error in each iteration conditioned on estimation errors in all preceding iterations.…”

“…In order to resolve lack of a rigorous proof, approximate message-passing (AMP) was proposed in [11] and proved in [12] to achieve the optimal performance for i.i.d. Gaussian measurements, when the compression rate is larger than the BP threshold.…”

Statistical estimation: from denoising to sparse regression and hidden cliques

“…We will sketch the main ideas referring to [16] for the original idea, to [17,41,42] for the analysis of the LASSO, and to [7,43,44] for extensions. This approach was also used in [45] to establish universality of the compressed sensing phase transition for non-Gaussian i.i.d.…”

“…(2) Derive an exact asymptotic characterization of the same algorithm as n, p → ∞, for t fixed. The characterization is given in terms of the so-called state evolution method developed rigorously in [41] (with generalizations in [43,45]). (3) Prove that AMP converges fast to the optimized θ, namely with high probability as n, p → ∞ we have θ (t) − θ 2 2 /p ≤ c 1 , e −c 2 t , with c 1 , c 2 two dimensionindependent constants.…”

“…We next carry out a heuristic analysis of AMP, referring to [41] for a rigorous treatment that uses ideas developed by Bolthausen in the context of mean-field spin glasses [55].…”