Selection Consistency of Lasso-Based Procedures for Misspecified High-Dimensional Binary Model and Random Regressors

Abstract

We consider selection of random predictors for high-dimensional regression problem with binary response for a general loss function. Important special case is when the binary model is semiparametric and the response function is misspecified under parametric model fit. Selection for such a scenario aims at recovering the support of the minimizer of the associated risk with large probability. We propose a two-step selection procedure which consists of screening and ordering predictors by Lasso method and then selecting a subset of predictors which minimizes Generalized Information Criterion on the corresponding nested family of models. We prove consistency of the selection method under conditions which allow for much larger number of predictors than number of observations. For the semiparametric case when distribution of random predictors satisfies linear regression conditions the true and the estimated parameters are collinear and their common support can be consistently identified.

Prediction and Variable Selection in High-Dimensional Misspecified Binary Classification

“…Subgaussianity of predictors is a standard assumption while working with random predictors in high-dimensional models, cf. [ 13 ]. In particular, Assumption 1 implies that and [ 29 ].…”

“…The latter has a nice information-theoretic interpretation. Namely, it can be viewed as the Kullback–Leibler projection of unknown on logistic models [ 13 ]. The Kullback–Leibler divergence [ 14 ] plays an important role in the information theory and statistics, for instance it is involved in information criteria in model selection [ 15 ] or in detecting influenctial observations [ 16 ].…”

“…It is an important problem and was studied thoroughly, among others in [ 17 , 18 , 22 , 23 , 24 ]. The variable selection and predictive properties of estimators in the high-dimensional scenario were studied, for instance, in [ 2 , 10 , 13 , 25 , 26 ]. In the current paper we investigate the behaviour of classifiers in possibly misspecified high-dimensional classification, which appears frequently in practice.…”

“…These two approaches to classification sometimes give unexpectedly good results in variable selection and prediction, but the reason of this phenomenon has not been deeply studied in the literature. Among the above mentioned papers only [ 2 , 13 , 25 ] take up this issue. However, [ 25 ] focuses mainly on the predictive properties of Lasso classifiers with the hinge loss.…”

“…However, [ 25 ] focuses mainly on the predictive properties of Lasso classifiers with the hinge loss. Bühlmann and van de Geer [ 2 ] and Kubkowski and Mielniczuk [ 13 ] study general Lipschitz loss functions. The latter paper considers only the variable selection problem.…”