A unifying tutorial on Approximate Message Passing

Feng, Oliver Y.; Venkataramanan, Ramji; Rush, Cynthia; Samworth, Richard J.

doi:10.48550/arxiv.2105.02180

Cited by 4 publications

(4 citation statements)

References 96 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The AMP algorithm and machinery has been successfully applied to a variety of problems beyond compressed sensing, including but not limited to robust M-estimators (Donoho and Montanari, 2016), SLOPE (Bu et al, 2020), low-rank matrix estimation and PCA (Rangan and Fletcher, 2012;Montanari and Venkataramanan, 2021;Fan, 2020;Zhong et al, 2021), stochastic block models (Deshpande et al, 2015), phase retrieval (Ma et al, 2018), phase synchronization (Celentano et al, 2021), and generalized linear models (Rangan, 2011;Barbier et al, 2019). See Feng et al (2021) for an accessible introduction of this machinery and its applications. Moreover, a dominant fraction of the AMP works focused on high-dimensional asymptotics (so that the problem dimension tends to infinity first before the number of iterations), except for Rush and Venkataramanan (2018) that derived finite-sample guarantees allowing the number of iterations to grow up to O(log n/ log log n).…”

Section: Approximate Message Passingmentioning

confidence: 99%

Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

Li¹,

Wei²

2021

Preprint

View full text Add to dashboard Cite

An evolving line of machine learning works observe empirical evidence that suggests interpolating estimators -the ones that achieve zero training error -may not necessarily be harmful. This paper pursues theoretical understanding for an important type of interpolators: the minimum 1-norm interpolator, which is motivated by the observation that several learning algorithms favor low 1-norm solutions in the over-parameterized regime. Concretely, we consider the noisy sparse regression model under Gaussian design, focusing on linear sparsity and high-dimensional asymptotics (so that both the number of features and the sparsity level scale proportionally with the sample size).We observe, and provide rigorous theoretical justification for, a curious multi-descent phenomenon; that is, the generalization risk of the minimum 1-norm interpolator undergoes multiple (and possibly more than two) phases of descent and ascent as one increases the model capacity. This phenomenon stems from the special structure of the minimum 1-norm interpolator as well as the delicate interplay between the over-parameterized ratio and the sparsity, thus unveiling a fundamental distinction in geometry from the minimum 2-norm interpolator. Our finding is built upon an exact characterization of the risk behavior, which is governed by a system of two non-linear equations with two unknowns.

show abstract

Section: Approximate Message Passingmentioning

confidence: 99%

Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

Li¹,

Wei²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Approximate Message Passing (AMP) belongs to the family of iterative thresholding algorithms and was initially conceived for compressed sensing applications [21]. Since then, variations of the algorithm have been applied to various statistical estimation tasks such as machine learning, image processing, and communications [22]. Based on this, the authors in [13], [23] propose the LArge MIMO AMP (LAMA) algorithm which is shown in Algorithm 1.…”

Section: B Approximate Message Passingmentioning

confidence: 99%

LAMANet: A Real-Time, Machine Learning-Enhanced Approximate Message Passing Detector for Massive MIMO

Brennsteiner

Arslan

Thompson

et al. 2023

IEEE Trans. VLSI Syst.

View full text Add to dashboard Cite

Model-driven machine learning for signal detection in the physical layer of mobile communication systems combines well-known detector structures with learned parameters. Recent work has shown high detection performance in massive Multiple-Input Multiple-Output (MIMO) detection, however, thorough complexity analysis and real-time processing hardware are lacking. This work proposes a novel machine learning enhanced Approximate Message Passing (AMP) algorithm named LAMANet and its hardware implementation. The algorithm solves some major challenges of previous proposals such as the complete loss of performance in untrained detectors and the still high computational complexity as compared to traditional massive MIMO detection methods. We provide a comprehensive complexity comparison, simulations of the Symbol Error Rate (SER) performance over realistic channel models, and a Field Programmable Gate Array (FPGA) implementation capable of processing LAMANet in real-time. The results show a similar detection performance of LAMANet to previous machine learningenhanced algorithms, while the computational effort is reduced to a level where real-time computation in hardware becomes comparable to traditional detection methods.

show abstract

“…First developed for Bayesian linear regression and compressed sensing in [26][27][28]35], they have since been applied to many high-dimensional problems arising in statistics and machine learning, including Lasso estimation and sparse linear regression [4,39], generalized linear models and phase retrieval [52,56,60], robust linear regression [24], sparse or structured principal components analysis (PCA) [22,23,44,53], group synchronization problems [51], deep learning [12,13,40] and optimization in spin glass models [1,32,42]. We refer to [30] for a recent review.…”

Section: Introductionmentioning

confidence: 99%

Approximate Message Passing algorithms for rotationally invariant matrices

Fan

2022

Ann. Statist.

View full text Add to dashboard Cite

Approximate Message Passing (AMP) algorithms have seen widespread use across a variety of applications. However, the precise forms for their Onsager corrections and state evolutions depend on properties of the underlying random matrix ensemble, limiting the extent to which AMP algorithms derived for white noise may be applicable to data matrices that arise in practice.In this work, we study more general AMP algorithms for random matrices W that satisfy orthogonal rotational invariance in law, where W may have a spectral distribution that is different from the semicircle and Marcenko-Pastur laws characteristic of white noise. The Onsager corrections and state evolutions in these algorithms are defined by the free cumulants or rectangular free cumulants of the spectral distribution of W. Their forms were derived previously by Opper, Çakmak and Winther using nonrigorous dynamic functional theory techniques, and we provide rigorous proofs.Our motivating application is a Bayes-AMP algorithm for Principal Components Analysis, when there is prior structure for the principal components (PCs) and possibly nonwhite noise. For sufficiently large signal strengths and any non-Gaussian prior distributions for the PCs, we show that this algorithm provably achieves higher estimation accuracy than the sample PCs.

show abstract

A unifying tutorial on Approximate Message Passing

Cited by 4 publications

References 96 publications

Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

LAMANet: A Real-Time, Machine Learning-Enhanced Approximate Message Passing Detector for Massive MIMO

Approximate Message Passing algorithms for rotationally invariant matrices

Contact Info

Product

Resources

About