Slope meets Lasso: Improved oracle bounds and optimality

Bellec, Pierre; Lecué, Guillaume; Tsybakov, Alexandre B.

doi:10.1214/17-aos1670

Cited by 126 publications

(190 citation statements)

References 22 publications

Supporting

Mentioning

184

Contrasting

Order By: Relevance

“…Finally, we show that solution to (6) and (7) is unique. From the proof of Lemma 24, equations (6), (7) correspond to the first order optimality condition of min σ≥0 max θ≥0 Ψ(σ, θ) and statonary point (σ * , θ * ) always exists.…”

Section: ) Proof Of Theoremmentioning

confidence: 81%

“…It is not easy to handle them simultaneously. A simplification we can make is to first set τ to 1 and find the minimum σ such that the first equation (6) and the inequality E B,Z [η (B + σZ; F y , F λ )] ≤ δ hold. Once we get σ min and optimal λ * , the corresponding τ min can then be obtained via (7): τ min = (1 − 1 δ E B,Z [η * (B + σ min Z; F y , F * λ )]) −1 and λ * is in turn updated to be λ * /τ min .…”

Section: F Proof Of Propositionmentioning

confidence: 99%

See 1 more Smart Citation

Asymptotics and Optimal Designs of SLOPE for Sparse Linear Regression

2019

2019 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

In sparse linear regression, the SLOPE estimator generalizes LASSO by assigning magnitude-dependent regularizations to different coordinates of the estimate. In this paper, we present an asymptotically exact characterization of the performance of SLOPE in the high-dimensional regime where the number of unknown parameters grows in proportion to the number of observations. Our asymptotic characterization enables us to derive optimal regularization sequences to either minimize the MSE or to maximize the power in variable selection under any given level of Type-I error. In both cases, we show that the optimal design can be recast as certain infinite-dimensional convex optimization problems, which have efficient and accurate finite-dimensional approximations. Numerical simulations verify our asymptotic predictions. They also demonstrate the superiority of our optimal design over LASSO and a regularization sequence previously proposed in the literature. DRAFT

show abstract

Section: ) Proof Of Theoremmentioning

confidence: 81%

Section: F Proof Of Propositionmentioning

confidence: 99%

Asymptotics and Optimal Designs of SLOPE for Sparse Linear Regression

2019

2019 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

show abstract

“…One example of such an order-optimal estimator is the SLOPE estimator which was recently analyzed [55]. If one is willing to tolerate a slightly slower rate of O σ 2 · k ln d n for signal recovery, several other estimators can be used.…”

Section: Side Informationmentioning

confidence: 99%

Harmless Interpolation of Noisy Data in Regression

Muthukumar¹,

Vodrahalli²,

Subramanian³

et al. 2020

IEEE J. Sel. Areas Inf. Theory

View full text Add to dashboard Cite

A continuing mystery in understanding the empirical success of deep neural networks is their ability to achieve zero training error and generalize well, even when the training data is noisy and there are more parameters than data points. We investigate this overparameterized regime in linear regression, where all solutions that minimize training error interpolate the data, including noise. We characterize the fundamental generalization (mean-squared) error of any interpolating solution in the presence of noise, and show that this error decays to zero with the number of features. Thus, overparameterization can be explicitly beneficial in ensuring harmless interpolation of noise. We discuss two root causes for poor generalization that are complementary in nature -signal "bleeding" into a large number of alias features, and overfitting of noise by parsimonious feature selectors. For the sparse linear model with noise, we provide a hybrid interpolating scheme that mitigates both these issues and achieves order-optimal MSE over all possible interpolating solutions. arXiv:1903.09139v2 [cs.LG] 9 Sep 2019 2. We provide a Fourier-theoretic interpretation of concurrent analyses [6-10] of the minimum 2 -norm interpolator.3. We show (Theorem 2) that parsimonious interpolators (like the 1 -minimizing interpolator and its relatives) suffer the complementary problem of overfitting pure noise.4. We construct two-step hybrid interpolators that successfully recover signal and harmlessly fit noise, achieving the order-optimal rate of test MSE among all interpolators (Proposition 1 and all its corollaries). Related workWe discuss prior work in three categories: a) overparameterization in deep neural networks, b) interpolation of high-dimensional data using kernels, and c) high-dimensional linear regression. We then recap work on overparameterized linear regression that is concurrent to ours. Recent interest in overparameterizationConventional statistical wisdom is that using more parameters in one's model than data points leads to poor generalization. This wisdom is corroborated in theory by worst-case generalization bounds on such overparameterized models following from VC-theory in classification [2] and ill-conditioning in least-squares regression [5]. It is, however, contradicted in practice by the notable recent trend of empirically successful overparameterized deep neural networks. For example, the commonly used CIFAR-10 dataset contains 60000 images, but the number of parameters in all the neural networks achieving state-of-the-art performance on CIFAR-10 is at least 1.5 million [4]. These neural networks have the ability to memorize pure noisesomehow, they are still able to generalize well when trained with meaningful data.Since the publication of this observation [4,11], the machine learning community has seen a flurry of activity to attempt to explain this phenomenon, both for classification and regression problems, in neural networks. The problem is challenging for three core reasons 2 :1. The optimization landscape for l...

show abstract

“…This paper is motivated by the SLOPE method [21] for its use of the SL1 regularization, where it brings many benefits not available with the popular 1 -based regularization-the capability of false discovery rate (FDR) control [21,22], adaptivity to unknown signal sparsity [23] and clustering of coefficients [24,25]. Also, efficient optimization methods [13,21,26] and more theoretical analysis [23,[27][28][29] are under active research.…”

Section: Lasso and Slopementioning

confidence: 99%

Structure Learning of Gaussian Markov Random Fields with False Discovery Rate Control

2019

View full text Add to dashboard Cite

In this paper, we propose a new estimation procedure for discovering the structure of Gaussian Markov random fields (MRFs) with false discovery rate (FDR) control, making use of the sorted 1 -norm (SL1) regularization. A Gaussian MRF is an acyclic graph representing a multivariate Gaussian distribution, where nodes are random variables and edges represent the conditional dependence between the connected nodes. Since it is possible to learn the edge structure of Gaussian MRFs directly from data, Gaussian MRFs provide an excellent way to understand complex data by revealing the dependence structure among many inputs features, such as genes, sensors, users, documents, etc. In learning the graphical structure of Gaussian MRFs, it is desired to discover the actual edges of the underlying but unknown probabilistic graphical model-it becomes more complicated when the number of random variables (features) p increases, compared to the number of data points n. In particular, when p n, it is statistically unavoidable for any estimation procedure to include false edges. Therefore, there have been many trials to reduce the false detection of edges, in particular, using different types of regularization on the learning parameters. Our method makes use of the SL1 regularization, introduced recently for model selection in linear regression. We focus on the benefit of SL1 regularization that it can be used to control the FDR of detecting important random variables. Adapting SL1 for probabilistic graphical models, we show that SL1 can be used for the structure learning of Gaussian MRFs using our suggested procedure nsSLOPE (neighborhood selection Sorted L-One Penalized Estimation), controlling the FDR of detecting edges.

show abstract

Slope meets Lasso: Improved oracle bounds and optimality

Cited by 126 publications

References 22 publications

Asymptotics and Optimal Designs of SLOPE for Sparse Linear Regression

Asymptotics and Optimal Designs of SLOPE for Sparse Linear Regression

Harmless Interpolation of Noisy Data in Regression

Structure Learning of Gaussian Markov Random Fields with False Discovery Rate Control

Contact Info

Product

Resources

About