Model selection with lasso-zero: adding straw to the haystack to better find needles

Abstract

The high-dimensional linear model y = Xβ 0 + is considered and the focus is put on the problem of recovering the support S 0 of the sparse vector β 0 . We introduce Lasso-Zero, a new 1 -based estimator whose novelty resides in an "overfit, then threshold" paradigm and the use of noise dictionaries concatenated to X for overfitting the response. To select the threshold, we employ the quantile universal threshold based on a pivotal statistic that requires neither knowledge nor preliminary estimation of the noise level. Numerical simulations show that Lasso-Zero performs well in terms of support recovery and provides an excellent trade-off between high true positive rate and low false discovery rate compared to competitors. Our methodology is supported by theoretical results showing that when no noise dictionary is used, Lasso-Zero recovers the signs of β 0 under weaker conditions on X and S 0 than the Lasso and achieves sign consistency for correlated Gaussian designs. The use of noise dictionary improves the procedure for low signals.

On the sign recovery by LASSO, thresholded LASSO and thresholded Basis Pursuit Denoising

“…BP estimator) in the full null model when β = 0 [18]. For the BP estimator, Descloux and Sardy [11] suggest the threshold τ fn α defined as the 1 − α quantile of max β fn 1 , . .…”

“…When entries of X are i.i.d N (0, 1), the optimal value of λ selected by AMP theory provides a thresholded LASSO for which the derived sign estimator is the best one to recover S(β). One may notice that the threshold selection provided in Descloux and Sardy [11] does not allow to recover S(β) with a large probability when β has lot of large components (intuitively when β is far from 0). Instead, our heuristic application of the knockoff methodology allows for almost perfect control of FWER at level 0.05.…”

“…These two expressions are given in Bühlmann and van de Geer [5] page 15 or in the proof of Theorem 1 of Zou [32]. Using equality (10) and inequality (11), we are going to show that if S( β L (λ)) = S(β) then the following event holds…”

Aggregated hold out for sparse linear regression with a robust loss function

“…In practice too, there is reason to consider alternatives to CV-based hyperparameter selection in sparse regression: sparse estimators are unstable, and selecting only one estimator can result in arbitrarily ignoring certain variables among a correlated group with similar predictive power [37]. For the Lasso, these difficulties have motivated researchers to introduce several aggregation schemes, such as the Bolasso [3], stability selection [19], the lasso-zero [9] and the random lasso [34], which are shown to have some better properties than the standard Lasso.…”

Robust Lasso-Zero for sparse corruption and model selection with missing covariates

“…

We propose Robust Lasso-Zero, an extension of the Lasso-Zero methodology [Descloux and Sardy, 2018], initially introduced for sparse linear models, to the sparse corruptions problem. We give theoretical guarantees on the sign recovery of the parameters for a slightly simplified version of the estimator, called Thresholded Justice Pursuit.

…”