Thresholded Basis Pursuit: LP Algorithm for Order-Wise Optimal Support Recovery for Sparse and Approximately Sparse Signals From Noisy Random Measurements

Saligrama, Venkatesh; Zhao, Manqi

doi:10.1109/tit.2011.2104512

Cited by 27 publications

(27 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We can now apply Laplace's principle in the previous section to prove (69). We begin by examining the pointwise convergence of the PMMSE estimator x u (y).…”

Section: Appendix C Large Deviations Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Asymptotic Analysis of MAP Estimation via the Replica Method and Applications to Compressed Sensing

Rangan

Fletcher

Goyal

2012

IEEE Trans. Inform. Theory

173

298

View full text Add to dashboard Cite

Abstract-The replica method is a non-rigorous but wellknown technique from statistical physics used in the asymptotic analysis of large, random, nonlinear problems. This paper applies the replica method, under the assumption of replica symmetry, to study estimators that are maximum a posteriori (MAP) under a postulated prior distribution. It is shown that with random linear measurements and Gaussian noise, the replica-symmetric prediction of the asymptotic behavior of the postulated MAP estimate of an n-dimensional vector "decouples" as n scalar postulated MAP estimators. The result is based on applying a hardening argument to the replica analysis of postulated posterior mean estimators of Tanaka and of Guo and Verdú.The replica-symmetric postulated MAP analysis can be readily applied to many estimators used in compressed sensing, including basis pursuit, lasso, linear estimation with thresholding, and zero norm-regularized estimation. In the case of lasso estimation the scalar estimator reduces to a soft-thresholding operator, and for zero norm-regularized estimation it reduces to a hardthreshold. Among other benefits, the replica method provides a computationally-tractable method for precisely predicting various performance metrics including mean-squared error and sparsity pattern recovery probability.

show abstract

“…We can now apply Laplace's principle in the previous section to prove (69). We begin by examining the pointwise convergence of the PMMSE estimator x u (y).…”

Section: Appendix C Large Deviations Resultsmentioning

confidence: 99%

“…This idea of using thresholding for sparsity detection has been proposed in [55] and [69]. Using the joint distribution (x j , s j ,x j ), one can then compute the probability of sparsity misdetection…”

Section: Support Recovery With Thresholdingmentioning

confidence: 99%

Asymptotic Analysis of MAP Estimation via the Replica Method and Applications to Compressed Sensing

Rangan

Fletcher

Goyal

2012

IEEE Trans. Inform. Theory

173

298

View full text Add to dashboard Cite

show abstract

“…The vast improvement of these estimators over the minimum 2 -norm interpolator is not surprising on some level, as at the very least these estimators will preserve signal. The estimator that uses BP in its first step, and then thresholds the coefficients that are below (half the) minimum absolute value of the non-zero coefficients of α * , has been shown to recover the sign pattern, or the true support, of α * exactly [54]. Indeed, this implies that after the first step, the true signal components (corresponding to entries in supp(α * )) are guaranteed to be preserved by BP even in the presence of low-enough levels of noise.…”

Section: Basis Pursuit (Bp)mentioning

confidence: 99%

Harmless Interpolation of Noisy Data in Regression

Muthukumar¹,

Vodrahalli²,

Subramanian³

et al. 2020

IEEE J. Sel. Areas Inf. Theory

View full text Add to dashboard Cite

A continuing mystery in understanding the empirical success of deep neural networks is their ability to achieve zero training error and generalize well, even when the training data is noisy and there are more parameters than data points. We investigate this overparameterized regime in linear regression, where all solutions that minimize training error interpolate the data, including noise. We characterize the fundamental generalization (mean-squared) error of any interpolating solution in the presence of noise, and show that this error decays to zero with the number of features. Thus, overparameterization can be explicitly beneficial in ensuring harmless interpolation of noise. We discuss two root causes for poor generalization that are complementary in nature -signal "bleeding" into a large number of alias features, and overfitting of noise by parsimonious feature selectors. For the sparse linear model with noise, we provide a hybrid interpolating scheme that mitigates both these issues and achieves order-optimal MSE over all possible interpolating solutions. arXiv:1903.09139v2 [cs.LG] 9 Sep 2019 2. We provide a Fourier-theoretic interpretation of concurrent analyses [6-10] of the minimum 2 -norm interpolator.3. We show (Theorem 2) that parsimonious interpolators (like the 1 -minimizing interpolator and its relatives) suffer the complementary problem of overfitting pure noise.4. We construct two-step hybrid interpolators that successfully recover signal and harmlessly fit noise, achieving the order-optimal rate of test MSE among all interpolators (Proposition 1 and all its corollaries). Related workWe discuss prior work in three categories: a) overparameterization in deep neural networks, b) interpolation of high-dimensional data using kernels, and c) high-dimensional linear regression. We then recap work on overparameterized linear regression that is concurrent to ours. Recent interest in overparameterizationConventional statistical wisdom is that using more parameters in one's model than data points leads to poor generalization. This wisdom is corroborated in theory by worst-case generalization bounds on such overparameterized models following from VC-theory in classification [2] and ill-conditioning in least-squares regression [5]. It is, however, contradicted in practice by the notable recent trend of empirically successful overparameterized deep neural networks. For example, the commonly used CIFAR-10 dataset contains 60000 images, but the number of parameters in all the neural networks achieving state-of-the-art performance on CIFAR-10 is at least 1.5 million [4]. These neural networks have the ability to memorize pure noisesomehow, they are still able to generalize well when trained with meaningful data.Since the publication of this observation [4,11], the machine learning community has seen a flurry of activity to attempt to explain this phenomenon, both for classification and regression problems, in neural networks. The problem is challenging for three core reasons 2 :1. The optimization landscape for l...

show abstract

“…where ε is a standard Gaussian random variable, ζ ∼ N (0, I n ) is a standard Gaussian random vector in R n independent of ε, and t(·) is defined in (10).…”

Section: Non-asymptotic Bounds On the Minimax Riskmentioning

confidence: 99%

Optimal Variable Selection and Adaptive Noisy Compressed Sensing

Ndaoud

Tsybakov

2020

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

For high-dimensional linear regression model, we propose an algorithm of exact support recovery in the setting of noisy compressed sensing where all entries of the design matrix are i.i.d standard Gaussian. This algorithm achieves the same conditions of exact recovery as the exhaustive search (maximal likelihood) decoder, and has an advantage over the latter of being adaptive to all parameters of the problem and computable in polynomial time.The core of our analysis consists in the study of the non-asymptotic minimax Hamming risk of variable selection. This allows us to derive a procedure, which is nearly optimal in a non-asymptotic minimax sense. Then, we develop its adaptive version, and propose a robust variant of the method to handle datasets with outliers and heavy-tailed distributions of observations. The resulting polynomial time procedure is near optimal, adaptive to all parameters of the problem and also robust.

show abstract

Thresholded Basis Pursuit: LP Algorithm for Order-Wise Optimal Support Recovery for Sparse and Approximately Sparse Signals From Noisy Random Measurements

Cited by 27 publications

References 27 publications

Asymptotic Analysis of MAP Estimation via the Replica Method and Applications to Compressed Sensing

Asymptotic Analysis of MAP Estimation via the Replica Method and Applications to Compressed Sensing

Harmless Interpolation of Noisy Data in Regression

Optimal Variable Selection and Adaptive Noisy Compressed Sensing

Contact Info

Product

Resources

About