Strong Rules for Discarding Predictors in Lasso-Type Problems

Abstract

Summary We consider rules for discarding predictors in lasso regression and related problems, for computational efficiency. El Ghaoui and his colleagues have propose ‘SAFE’ rules, based on univariate inner products between each predictor and the outcome, which guarantee that a coefficient will be 0 in the solution vector. This provides a reduction in the number of variables that need to be entered into the optimization. We propose strong rules that are very simple and yet screen out far more predictors than the SAFE rules. This great practical improvement comes at a price: the strong rules are not foolproof and can mistakenly discard active predictors, i.e. predictors that have non-zero coefficients in the solution. We therefore combine them with simple checks of the Karush–Kuhn–Tucker conditions to ensure that the exact solution to the convex problem is delivered. Of course, any (approximate) screening method can be combined with the Karush–Kuhn–Tucker, conditions to ensure the exact solution; the strength of the strong rules lies in the fact that, in practice, they discard a very large number of the inactive predictors and almost never commit mistakes. We also derive conditions under which they are foolproof. Strong rules provide substantial savings in computational time for a variety of statistical optimization problems.

Development and validation of Risk Equations for Complications Of type 2 Diabetes (RECODe) using individual participant data from randomised trials

“…Elastic net regularisation is a machine learning approach designed to select models in the context of collinearity, which often leads to unstable estimates from traditional stepwise selection approaches; 18,19 this approach fits a Cox model via penalised maximum likelihood, using 10-times internal cross-validation to minimise the risk of overfitting. We did not include time-varying covariates because the risk equations are intended for use in clinical settings to assist initial treatment decisions.…”

A fast unified algorithm for solving group-lasso penalize learning problems

“…This subset is referred as the active-set. The similar idea has been adopted by previous work (Tibshirani et al 2012;Meier et al 2008;Vogt and Roth 2012). In detail, we first create an active-set A by updating each group belonging to S once.…”

“…For computing the solution at each λ we also utilize the strong rule introduced in Tibshirani et al (2012). Suppose that we have computed β(λ [l] ), the solution at λ [l] .…”

Tikhonov or Lasso Regularization: Which Is Better and When

“…A contribution of our work is to show the consequences of this property on the relative accuracy of the 1 and 2 solutions. Using SafeRule as a motivation, Tibishirani et al [12] has proposed Strong Rules which are more aggressive in pruning features. However, they are known to lead to inconsistent models at least in theoretically constructed cases.…”