Discussion: Deterioration of performance of the lasso with many predictors

Abstract

Oracle inequalities provide probability loss bounds for the lasso estimator at a deterministic choice of the regularization parameter and are commonly cited as theoretical justification for the lasso and its ability to handle high-dimensional settings. Unfortunately, in practice, the regularization parameter is not selected to be a deterministic quantity, but is instead chosen using a random, data-dependent procedure, often making these inequalities misleading in their implications. We discuss general results and demonstrate empirically for data using categorical predictors that the amount of deterioration in performance of the lasso as the number of unnecessary predictors increases can be far worse than the oracle inequalities suggest, but imposing structure on the form of the estimates can reduce this deterioration substantially.We would like to congratulate Gerhard Tutz and Jan Gertheiss (hereafter TG) on their impressive tour through the world of regularized regression for categorical data. Categorical predictors arise often in practice, bringing with them problems of potentially overspecified models that cannot be well-fit without the imposition of additional structure in the model. Regularization methods provide a natural way to impose such structure, as TG demonstrate.