Sparsity oracle inequalities for the Lasso

Bunea, Florentina; Tsybakov, Alexandre B.; Wegkamp, Marten

doi:10.1214/07-ejs008

Cited by 357 publications

(427 citation statements)

References 26 publications

Supporting

Mentioning

420

Contrasting

Order By: Relevance

“…Motivated by many practical prediction problems, including those that arise in microarray data analysis and natural language processing, this problem has been extensively studied in recent years. The results can be divided into two categories: those that study the predictive power ofβ [9,30,12] and those that study its sparsity pattern and reconstruction properties [4,32,18,19,17,8]; this article falls into the first of these categories.…”

Section: Introductionmentioning

confidence: 99%

ℓ 1-regularized linear regression: persistence and oracle inequalities

Bartlett

Mendelson

Neeman

2011

Probab. Theory Relat. Fields

View full text Add to dashboard Cite

We study the predictive performance of 1 -regularized linear regression, including the case where the number of covariates is substantially larger than the sample size. We introduce a new analysis method that does not require uniformly bounded covariates, an assumption that was often necessary with previous techniques. This technique provides an answer to a conjecture of Greenshtein and Ritov [12] regarding the "persistence" rate for linear regression and allows us to prove an oracle inequality for the error of the regularized minimizer.

show abstract

Section: Introductionmentioning

confidence: 99%

ℓ 1-regularized linear regression: persistence and oracle inequalities

Bartlett

Mendelson

Neeman

2011

Probab. Theory Relat. Fields

View full text Add to dashboard Cite

show abstract

“…The standard choice of λ employs 5) where A ≥ 1 is a constant that does not depend on X, chosen so that (2.4) holds no matter what X is. Note that √ n S ∞ is a maximum of N (0, σ 2 ) variables, which are correlated if columns of X are correlated, as they typically are in the sample.…”

Section: Lasso As a Benchmark In Parametric And Nonparametric Modelsmentioning

confidence: 99%

“…Thus the estimator can be consistent and can have excellent forecasting performance even under very rapid, nearly exponential growth of the total number of regressors p. [1] investigated the ℓ 1 -penalized quantile regression process, obtaining similar results. See [9,2,3,4,5,11,12,15] for many other interesting developments and a detailed review of the existing literature.…”

Section: Introductionmentioning

confidence: 99%

Post-l1-penalized estimators in high-dimensional linear regression models

Belloni¹,

Chernozhukov²

2010

View full text Add to dashboard Cite

Abstract. In this paper we study post-penalized estimators which apply ordinary, unpenalized linear regression to the model selected by first-step penalized estimators, typically LASSO.It is well known that LASSO can estimate the regression function at nearly the oracle rate, and is thus hard to improve upon. We show that post-LASSO performs at least as well as LASSO in terms of the rate of convergence, and has the advantage of a smaller bias. Remarkably, this performance occurs even if the LASSO-based model selection "fails" in the sense of missing some components of the "true" regression model. By the "true" model we mean here the best s-dimensional approximation to the regression function chosen by the oracle. Furthermore, post-LASSO can perform strictly better than LASSO, in the sense of a strictly faster rate of convergence, if the LASSO-based model selection correctly includes all components of the "true" model as a subset and also achieves a sufficient sparsity. In the extreme case, when LASSO perfectly selects the "true" model, the post-LASSO estimator becomes the oracle estimator. An important ingredient in our analysis is a new sparsity bound on the dimension of the model selected by LASSO which guarantees that this dimension is at most of the same order as the dimension of the "true" model. Our rate results are non-asymptotic and hold in both parametric and nonparametric models. Moreover, our analysis is not limited to the LASSO estimator in the first step, but also applies to other estimators, for example, the trimmed LASSO, Dantzig selector, or any other estimator with good rates and good sparsity. Our analysis covers both traditional trimming and a new practical, completely data-driven trimming scheme that induces maximal sparsity subject to maintaining a certain goodness-of-fit. The latter scheme has theoretical guarantees similar to those of LASSO or post-LASSO, but it dominates these procedures as well as traditional trimming in a wide variety of experiments.

show abstract

“…Theoretical properties of the lasso and related methods for high dimensional data have been examined by Fan and Peng (2004), Bunea et al (2007), Candès and Tao (2007), Huang et al (2008a,b), Kim et al (2008), Bickel et al (2009) and Meinshausen and Yu (2009), among many others. Most of the references consider quadratic objective functions and linear or nonparametric models with an additive mean 0 error.…”

Section: Introductionmentioning

confidence: 99%

The lasso for high-dimensional regression with a possible change-point

Lee¹,

Seo²,

Shin³

2014

View full text Add to dashboard Cite

Summary. We consider a high dimensional regression model with a possible change point due to a covariate threshold and develop the lasso estimator of regression coefficients as well as the threshold parameter. Our lasso estimator not only selects covariates but also selects a model between linear and threshold regression models. Under a sparsity assumption, we derive non-asymptotic oracle inequalities for both the prediction risk and the l 1 -estimation loss for regression coefficients. Since the lasso estimator selects variables simultaneously, we show that oracle inequalities can be established without pretesting the existence of the threshold effect. Furthermore, we establish conditions under which the estimation error of the unknown threshold parameter can be bounded by a factor that is nearly n 1 even when the number of regressors can be much larger than the sample size n. We illustrate the usefulness of our proposed estimation method via Monte Carlo simulations and an application to real data.

show abstract

Sparsity oracle inequalities for the Lasso

Cited by 357 publications

References 26 publications

ℓ 1-regularized linear regression: persistence and oracle inequalities

ℓ 1-regularized linear regression: persistence and oracle inequalities

Post-l1-penalized estimators in high-dimensional linear regression models

The lasso for high-dimensional regression with a possible change-point

Contact Info

Product

Resources

About