Aggregation and Sparsity Via ℓ1 Penalized Least Squares

Bunea, Florentina; Tsybakov, Alexandre B.; Wegkamp, Marten

doi:10.1007/11776420_29

Cited by 61 publications

(80 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For a discussion of the concept of SOI we refer to (Tsybakov 2006). Examples of SOI are proved in (Koltchinskii 2006;Bunea et al 2006Bunea et al , 2007aBunea et al , 2007bvan de Geer 2006;Bickel et al 2007) for the Lasso, BIC and Dantzig selector aggregates. Note that the SOI for the Lasso and Dantzig selector are not as strong as those for the BIC: they fail to guarantee optimal rates for MS, linear and convex aggregation unless φ 1 , .…”

Section: ))mentioning

confidence: 99%

Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity

2008

View full text Add to dashboard Cite

show abstract

Section: ))mentioning

confidence: 99%

Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity

2008

View full text Add to dashboard Cite

show abstract

“…Several papers have begun to investigate estimation of hdsms, primarily focusing on mean regression with the ℓ 1 -norm acting as a penalty function [4,6,7,8,9,17,22,28,31,33]. The results in [4,6,7,8,17,22,31,33] demonstrated the fundamental result that ℓ 1 -penalized least squares estimators achieve the rate s/n √ log p, which is very close to the oracle rate s/n achievable when the true model is known.…”

Section: Introductionmentioning

confidence: 94%

Least Squares After Model Selection in High-Dimensional Sparse Models

2009

View full text Add to dashboard Cite

Abstract. In this paper we study post-model selection estimators which apply ordinary least squares (ols) to the model selected by first-step penalized estimators, typically lasso. It is well known that lasso can estimate the nonparametric regression function at nearly the oracle rate, and is thus hard to improve upon. We show that ols post lasso estimator performs at least as well as lasso in terms of the rate of convergence, and has the advantage of a smaller bias. Remarkably, this performance occurs even if the lasso-based model selection "fails" in the sense of missing some components of the "true" regression model. By the "true" model we mean here the best s-dimensional approximation to the nonparametric regression function chosen by the oracle. Furthermore, ols post lasso estimator can perform strictly better than lasso, in the sense of a strictly faster rate of convergence, if the lasso-based model selection correctly includes all components of the "true" model as a subset and also achieves sufficient sparsity. In the extreme case, when lasso perfectly selects the "true" model, the ols post lasso estimator becomes the oracle estimator. An important ingredient in our analysis is a new sparsity bound on the dimension of the model selected by lasso which guarantees that this dimension is at most of the same order as the dimension of the "true" model. Our rate results are non-asymptotic and hold in both parametric and nonparametric models. Moreover, our analysis is not limited to the lasso estimator acting as selector in the first step, but also applies to any other estimator, for example various forms of thresholded lasso, with good rates and good sparsity properties. Our analysis covers both traditional thresholding and a new practical, data-driven thresholding scheme that induces maximal sparsity subject to maintaining a certain goodness-of-fit. The latter scheme has theoretical guarantees similar to those of lasso or ols post lasso, but it dominates these procedures as well as traditional thresholding in a wide variety of experiments.

show abstract

“…Readers are referred to Shao (1997) for more discussion on this issue. When p is allowed to increase with n, Bunea et al (2006) show that consistent variable selection can also be achieved via multiple testing. Much more general choices of k involving types of cross validation are given later in this section.…”

Section: Regressionmentioning

confidence: 99%

“…Note that, in fact, (2) can be proved for other procedures: a first example is given in Bunea et al (2005Bunea et al ( , 2006 where (2) is established for a Lasso typef n in the regression model with squared loss.…”

Section: Mannor Et Al (2003) Lugosi and Vayatismentioning

confidence: 99%

Regularization in statistics

Bickel

Tsybakov

et al. 2006

Test

Self Cite

189

126

View full text Add to dashboard Cite

This paper is a selective review of the regularization methods scattered in statistics literature. We introduce a general conceptual approach to regularization and fit most existing methods into it. We have tried to focus on the importance of regularization when dealing with today's high-dimensional objects: data and models. A wide range of examples are discussed, including nonparametric regression, boosting, covariance matrix estimation, principal component estimation, subsampling.

show abstract

Aggregation and Sparsity Via ℓ1 Penalized Least Squares

Cited by 61 publications

References 15 publications

Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity

Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity

Least Squares After Model Selection in High-Dimensional Sparse Models

Regularization in statistics

Contact Info

Product

Resources

About