Algorithmic Analysis and Statistical Estimation of SLOPE via Approximate Message Passing

Bu, Zhiqi; Klusowski, Jason M.; Rush, Cynthia; Su, Weijie

doi:10.1109/tit.2020.3025272

Cited by 28 publications

(24 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This state-evolution characterization made AMP amenable as a analysis device to describe the statistical behaviors for various problems, despite that AMP is an effective algorithm on its own. The AMP algorithm and machinery has been successfully applied to a variety of problems beyond compressed sensing, including but not limited to robust M-estimators (Donoho and Montanari, 2016), SLOPE (Bu et al, 2020), low-rank matrix estimation and PCA (Rangan and Fletcher, 2012;Montanari and Venkataramanan, 2021;Fan, 2020;Zhong et al, 2021), stochastic block models (Deshpande et al, 2015), phase retrieval (Ma et al, 2018), phase synchronization (Celentano et al, 2021), and generalized linear models (Rangan, 2011;Barbier et al, 2019). See Feng et al (2021) for an accessible introduction of this machinery and its applications.…”

Section: Approximate Message Passingmentioning

confidence: 99%

Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

Li¹,

Wei²

2021

Preprint

View full text Add to dashboard Cite

An evolving line of machine learning works observe empirical evidence that suggests interpolating estimators -the ones that achieve zero training error -may not necessarily be harmful. This paper pursues theoretical understanding for an important type of interpolators: the minimum 1-norm interpolator, which is motivated by the observation that several learning algorithms favor low 1-norm solutions in the over-parameterized regime. Concretely, we consider the noisy sparse regression model under Gaussian design, focusing on linear sparsity and high-dimensional asymptotics (so that both the number of features and the sparsity level scale proportionally with the sample size).We observe, and provide rigorous theoretical justification for, a curious multi-descent phenomenon; that is, the generalization risk of the minimum 1-norm interpolator undergoes multiple (and possibly more than two) phases of descent and ascent as one increases the model capacity. This phenomenon stems from the special structure of the minimum 1-norm interpolator as well as the delicate interplay between the over-parameterized ratio and the sparsity, thus unveiling a fundamental distinction in geometry from the minimum 2-norm interpolator. Our finding is built upon an exact characterization of the risk behavior, which is governed by a system of two non-linear equations with two unknowns.

show abstract

Section: Approximate Message Passingmentioning

confidence: 99%

Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

Li¹,

Wei²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Sharp asymptotic predictions have only more recently appeared in the literature for the case of noisy linear measurements with Gaussian measurement vectors. There are by now three different approaches that have been used towards asymptotic analysis of convex regularized estimators: (i) the one that is based on the approximate message passing (AMP) algorithm and its state-evolution analysis (e.g., [5,8,14,20,[32][33][34]); (ii) the one that is based on Gaussian process (GP) inequalities, specifically on the convex Gaussian min-max Theorem (CGMT) (e.g., [9,10,13,15,18,19]); and (iii) the "leave-one-out" approach [11,35]. The three approaches are quite different to each other and each comes with its unique distinguishing features and disadvantages.…”

Section: Related Workmentioning

confidence: 99%

“…Before stating the result, we provide some intuition about the proof which builds on Theorem 2. The critical observation in the proof of Theorem 2 is that the effective noise σ of is minimized (i.e., it attains the value σ opt ) if the Cauchy-Schwartz inequality in (20) holds with equality. Hence, we seek = opt so that for some c ∈ R,…”

Section: On the Optimal Loss Functionmentioning

confidence: 99%

“…The literature on the topic is vast with remarkable contributions from the statistics, signal processing and machine learning communities. Several recent works focus on the proportional/linear asymptotic regime and derive sharp results on the inference performance of appropriate convex optimization methods (e.g., [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ]). These works show that, albeit challenging, sharp results are advantageous over loose order-wise bounds.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Sharp Guarantees and Optimal Performance for Inference in Binary and Gaussian-Mixture Models

Taheri

Pedarsani

Thrampoulidis

2021

Entropy

View full text Add to dashboard Cite

We study convex empirical risk minimization for high-dimensional inference in binary linear classification under both discriminative binary linear models, as well as generative Gaussian-mixture models. Our first result sharply predicts the statistical performance of such estimators in the proportional asymptotic regime under isotropic Gaussian features. Importantly, the predictions hold for a wide class of convex loss functions, which we exploit to prove bounds on the best achievable performance. Notably, we show that the proposed bounds are tight for popular binary models (such as signed and logistic) and for the Gaussian-mixture model by constructing appropriate loss functions that achieve it. Our numerical simulations suggest that the theory is accurate even for relatively small problem dimensions and that it enjoys a certain universality property.

show abstract

“…This method relies on the computation of the proximal operator for the sorted l 1 norm. This operator is also a very important tool for the development of the approximate message passing theory (Bu et al, 2020;Zhang and Bu, 2021).…”

Section: Introductionmentioning

confidence: 99%