Finite-sample analysis of Approximate Message Passing

Abstract

Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing and low-rank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For concreteness, we consider the setting of high-dimensional regression, where the goal is to estimate a high-dimensional vector β 0 from a noisy measurement y = Aβ 0 + w. AMP is a low-complexity, scalable algorithm for this problem. Under suitable assumptions on the measurement matrix A, AMP has the attractive feature that its performance can be accurately characterized in the large system limit by a simple scalar iteration called state evolution. Previous proofs of the validity of state evolution have all been asymptotic convergence results. In this paper, we derive a concentration inequality for AMP with i.i.d. Gaussian measurement matrices with finite size n × N. The result shows that the probability of deviation from the state evolution prediction falls exponentially in n. This provides theoretical support for empirical findings that have demonstrated excellent agreement of AMP performance with state evolution predictions for moderately large dimensions. The concentration inequality also indicates that the number of AMP iterations t can grow no faster than order log n log log n for the performance to be close to the state evolution predictions with high probability. The analysis can be extended to obtain similar non-asymptotic results for AMP in other settings such as low-rank matrix estimation.

Message‐Passing Algorithms for Synchronization Problems over Compact Groups

“…It appears that proving this would require a nonasymptotic analysis of AMP, such as [58]. In practice, we initialize AMP to small random values.…”

“…Instead we would like to show that after !.1/ iterations we achieve a nonzero . It appears that proving this would require a nonasymptotic analysis of AMP, such as [58]. It may appear that this initialization issue can be fixed by initializing AMP with a spectral method, which achieves .1/ correlation with the truth; however, this does not appear to work easily due to a subtle issue about correlation between the noise and iterates.…”

Analysis of Approximate Message Passing With Non-Separable Denoisers and Markov Random Field Priors

“…the center coordinate of the first argument. In the context of AMP, as made explicit in (25), the termŝ h t+1 andq t measure the error in the observation V −1 (A * z t ) + β t and the estimate β t at time t, respectively, (the error w.r.t. the true β).…”

A Simple Derivation of AMP and its State Evolution via First-Order Cancellation

“…Gaussian A [4,5] as m, n → ∞ with m/n → δ ∈ (0, ∞). More recently, it has been proven that the state-evolution accurately characterizes AMP's behavior for large but finite m, n [6]. The rigorous SE proofs in [2,3,6], however, are long and complicated, and thus remain out of reach for many readers.…”

“…More recently, it has been proven that the state-evolution accurately characterizes AMP's behavior for large but finite m, n [6]. The rigorous SE proofs in [2,3,6], however, are long and complicated, and thus remain out of reach for many readers. And, although the AMP algorithm can be heuristically derived from an approximation of loop belief propagation (LBP), as outlined in [1] and [7], the LBP perspective is lacking in several respects.…”