Information criteria for variable selection under sparsity

Abstract

The optimization of an information criterion in a variable selection procedure leads to an additional bias, which can be substantial in sparse, high-dimensional data. The bias can be compensated by applying shrinkage while estimating within the selected models. This paper presents modified information criteria for use in variable selection and estimation without shrinkage. The analysis motivating the modified criteria follows two routes. The first, explored for signal-plus-noise observations only, goes by comparison of estimators with and without shrinkage. The second, discussed for general regression models, describes the optimization or selection bias as a double-sided effect, named a mirror effect in the paper: among the numerous insignificant variables, those with large, noisy values present themselves as being more valuable than an arbitrary variable, while in fact, they carry more noise than an arbitrary variable. The mirror effect is developed for Akaike's Information Criterion and for Mallows' C p , with special attention to the latter criterion as a stopping rule in a least angle regression routine. The result is a new stopping rule, not focusing on the quality of a lasso shrinkage selection, but on the least squares estimator without shrinkage within the same selection.

Sparse Regularization for Bi-Level Variable Selection

“…Zou et al (2007), Zhang et al (2010), Jansen (2014) and Janson et al (2015) discussed model selection criteria or degrees of freedom of the estimated model via L 1 type regularization. Breheny and Huang (2009) have derived the AIC (Akaike, 1974) and BIC (Schwarz, 1978) type criteria by defining the degrees of freedom of the estimated model via the group bridge, but those criteria are not well considered in other penalties for bi-level selection.…”

Model Selection and Model Averaging

“…Finding the best model for estimation without shrinkage is more delicate, as the prediction error for a misspecified l rapidly increases (Jansen, 2014). It thus determines the size of the selected model.…”

Asymptotic post-selection inference for the Akaike information criterion

“…For forward selection and least angle regression in normal linear regression models, Taylor et al (2016) study selective hypothesis tests and confidence intervals. Jansen (2014) studied the effect of the optimization on the expected values of the Akaike information criterion and Mallow's C p in high-dimensional sparse models. Belloni et al (2015) obtained uniformly valid confidence intervals in the presence of a sparse high-dimensional nuisance parameter.…”