Improving Lasso for model selection and prediction

Abstract

It is known that the Thresholded Lasso (TL), SCAD or MCP correct intrinsic estimation bias of the Lasso. In this paper we propose an alternative method of improving the Lasso for predictive models with general convex loss functions which encompass normal linear models, logistic regression, quantile regression or support vector machines. For a given penalty we order the absolute values of the Lasso non-zero coefficients and then select the final model from a small nested family by the Generalized Information Criterion. We derive exponential upper bounds on the selection error of the method. These results confirm that, at least for normal linear models, our algorithm seems to be the benchmark for the theory of model selection as it is constructive, computationally efficient and leads to consistent model selection under weak assumptions. Constructivity of the algorithm means that, in contrast to the TL, SCAD or MCP, consistent selection does not rely on the unknown parameters as the cone invertibility factor. Instead, our algorithm only needs the sample size, the number of predictors and an upper bound on the noise parameter. We show in numerical experiments on synthetic and real-world data sets that an implementation of our algorithm is more accurate than implementations of studied concave regularizations. Our procedure is contained in the R package ,,DMRnet" and available on the CRAN repository.

On the sign recovery by least absolute shrinkage and selection operator, thresholded least absolute shrinkage and selection operator, and thresholded basis pursuit denoising

“…It is well known that LASSO can consistently estimate 𝛽 under much weaker assumptions than the irrepresentability condition (see, e.g., Meinshausen & Yu, 2009) or (Van De Geer and Bühlmann 2009). This suggests that an appropriately thresholded version of LASSO can recover S(𝛽) under weaker assumptions than the irrepresentability condition (Pokarowski et al, 2019).…”

Robust Lasso‐Zero for sparse corruption and model selection with missing covariates

“…To ease the readability of the numerical results, we also provide in Appendix C.2 additional comparisons to other estimators: the thresholded Robust Lasso proposed in Nguyen and Tran (2013a) and the thresholded lasso in Pokarowski et al (2019). Rlass0 still remains competitive in terms of sign recovery, in particular in difficult cases, that is, when the percentage of missing values increases, when the missing data are informative and when the covariates are correlated.…”

“…For a non-oracle hyperparameter tuning, we compare the Robust Lasso-Zero with the thresholded Robust Lasso proposed in Nguyen and Tran (2013a) and the thresholded lasso in Pokarowski et al (2019).…”

“…For the thresholded version of the Robust-Lasso and of the Lasso, we used the Generalized Information Criterion (GIC), proposed in the R package DMRnet Pokarowski et al (2019). One could note that this criterion requires the estimation of the noise level, which is not needed for the Robust Lasso-Zero using QUT instead.…”