Universality of Approximate Message Passing with Semi-Random Matrices

Dudeja, Rishabh; Lu, Yue M.; Sen, Subhabrata

doi:10.48550/arxiv.2204.04281

Cited by 6 publications

(23 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…VAMP algorithms play a crucial role in this paper as well. At the heart of our results in the current paper is a universality principle for VAMP algorithms (Theorem 3), which generalizes the result obtained in our prior work in [37] in several important ways. First, we now allow VAMP algorithms to use some side information in their updates.…”

Section: Related Worksupporting

confidence: 77%

“…Universality of AMP for Sign and Permutation Invariant Matrices. After our earlier work [37] appeared on arXiv and during the preparation of the current manuscript, a parallel work of Wang et al…”

Section: Related Workmentioning

confidence: 82%

“…There have been rigorous studies of the universality phenomenon (see, e.g., [3,34,41,95]) in contexts related to Figure 1. We provide a detailed discussion of these earlier works in the literature as well as several recent advances [36,37,100] in Section 3.…”

Section: Universality Of Rls Estimatorsmentioning

confidence: 99%

“…AMP algorithms for Semi-Random Matrices. In recent work [37], the authors of this manuscript have obtained a universality principle for a subclass of non-linear AMP algorithms called vector approximate message passing (VAMP) algorithms [19,63,81,91]. This work identified a notion of semi-random matrices such that VAMP algorithms driven by any matrix in this class have the same limiting dynamics.…”

Section: Related Workmentioning

confidence: 99%

“…This feature is necessary to capture inference problems like regularized linear regression, where the signal and the noise are treated as side information. Furthermore, unlike in [37] where the VAMP algorithms are restricted to be memoryfree, we now allow VAMP algorithms to use the entire history of previous iterates in their update rules. This extension enables us to implement general first-order methods (like the proximal method) using VAMP algorithms and hence obtain a universality principle for the proximal method and, subsequently, the RLS estimator.…”

Section: Related Workmentioning

confidence: 99%

See 4 more Smart Citations

Spectral Universality of Regularized Linear Regression with Nearly Deterministic Sensing Matrices

Dudeja¹,

Sen²,

Lu³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

It has been observed that the performances of many high-dimensional estimation problems are universal with respect to underlying sensing (or design) matrices. Specifically, matrices with markedly different constructions seem to achieve identical performance if they share the same spectral distribution and have "generic" singular vectors. We prove this universality phenomenon for the case of convex regularized least squares (RLS) estimators under a linear regression model with additive Gaussian noise. Our main contributions are two-fold: (1) We introduce a notion of universality classes for sensing matrices, defined through a set of deterministic conditions that fix the spectrum of the sensing matrix and precisely capture the previously heuristic notion of generic singular vectors; (2) We show that for all sensing matrices that lie in the same universality class, the dynamics of the proximal gradient descent algorithm for solving the regression problem, as well as the performance of RLS estimators themselves (under additional strong convexity conditions) are asymptotically identical. In addition to including i.i.d. Gaussian and rotational invariant matrices as special cases, our universality class also contains highly structured, strongly correlated, or even (nearly) deterministic matrices. Examples of the latter include randomly signed versions of incoherent tight frames and randomly subsampled Hadamard transforms. As a consequence of this universality principle, the asymptotic performance of regularized linear regression on many structured matrices constructed with limited randomness can be characterized by using the rotationally invariant ensemble as an equivalent yet mathematically more tractable surrogate.

show abstract

Section: Related Worksupporting

confidence: 77%

Section: Related Workmentioning

confidence: 82%

Section: Universality Of Rls Estimatorsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Spectral Universality of Regularized Linear Regression with Nearly Deterministic Sensing Matrices

Dudeja¹,

Sen²,

Lu³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Exponentially Fast Concentration of Vector Approximate Message Passing to its State Evolution

Cademartori

Rush

2020

2020 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

Approximate Message Passing (AMP) algorithms are a class of iterative procedures for computationally-efficient estimation in high-dimensional inference and estimation tasks. Due to the presence of an 'Onsager' correction term in its iterates, for N × M design matrices A with i.i.d. Gaussian entries, the asymptotic distribution of the estimate at any iteration of the algorithm can be exactly characterized in the large system limit as M/N → δ ∈ (0, ∞) via a scalar recursion referred to as state evolution. In this paper, we show that appropriate functionals of the iterates, in fact, concentrate around their limiting values predicted by these asymptotic distributions with rates exponentially fast in N for a large class of AMP-style algorithms, including those that are used when high-dimensional generalized linear regression models are assumed to be the data-generating process, like the generalized AMP algorithm, or those that are used when the measurement matrix is assumed to be right rotationally invariant instead of i.i.d. Gaussian, like vector AMP and generalized vector AMP. In practice, these more general AMP algorithms have many applications, for example in in communications or imaging, and this work provides the first study of finite sample behavior of such algorithms. c 2k (k!) κ ′ with κ, κ ′ and c, c ′ being universal constants not depending on ǫ or N . While we do not specify κ, κ ′ explicitly in this informal statement, these will be small constant values (like κ = 16 and κ ′ = 22 from Theorem 1).

show abstract

A Non-Asymptotic Framework for Approximate Message Passing in Spiked Models

Li¹,

Wei²

2022

Preprint

View full text Add to dashboard Cite

Approximate message passing (AMP) emerges as an effective iterative paradigm for solving highdimensional statistical problems. However, prior AMP theory -which focused mostly on high-dimensional asymptotics -fell short of predicting the AMP dynamics when the number of iterations surpasses o log n log log n (with n the problem dimension). To address this inadequacy, this paper develops a nonasymptotic framework for understanding AMP in spiked matrix estimation. Built upon new decomposition of AMP updates and controllable residual terms, we lay out an analysis recipe to characterize the finitesample behavior of AMP in the presence of an independent initialization, which is further generalized to allow for spectral initialization. As two concrete consequences of the proposed analysis recipe: (i) when solving Z2 synchronization, we predict the behavior of spectrally initialized AMP for up to O n poly log n iterations, showing that the algorithm succeeds without the need of a subsequent refinement stage (as conjectured recently by Celentano et al. (2021)); (ii) we characterize the non-asymptotic behavior of AMP in sparse PCA (in the spiked Wigner model) for a broad range of signal-to-noise ratio.

show abstract

Universality of Approximate Message Passing with Semi-Random Matrices

Cited by 6 publications

References 42 publications

Spectral Universality of Regularized Linear Regression with Nearly Deterministic Sensing Matrices

Spectral Universality of Regularized Linear Regression with Nearly Deterministic Sensing Matrices

Exponentially Fast Concentration of Vector Approximate Message Passing to its State Evolution

A Non-Asymptotic Framework for Approximate Message Passing in Spiked Models

Contact Info

Product

Resources

About