Post selection shrinkage estimation for high‐dimensional data analysis

Abstract

In high-dimensional data settings where p n, many penalized regularization approaches were studied for simultaneous variable selection and estimation. However, with the existence of covariates with weak effect, many existing variable selection methods, including Lasso and its generations, cannot distinguish covariates with weak and no contribution. Thus, prediction based on a subset model of selected covariates only can be inefficient. In this paper, we propose a post selection shrinkage estimation strategy to improve the prediction performance of a selected subset model. Such a post selection shrinkage estimator (PSE) is data-adaptive and constructed by shrinking a post selection weighted ridge estimator in the direction of a selected candidate subset. Under an asymptotic distributional quadratic risk criterion, its prediction performance is explored analytically. We show that the proposed post selection PSE performs better than the post selection weighted ridge estimator. More importantly, it improves the prediction performance of any candidate subset model selected from most existing Lasso-type variable selection methods significantly. The relative performance of the post selection PSE is demonstrated by both simulation studies and real data analysis.

Weak signals in high‐dimensional regression: Detection, estimation and prediction

“…The following corollary shows that Theorem 3.1, together with Theorem 2 in Zhang and Huang (2008) and Corollary 2 in Gao et al (2017), further implies selection consistency for ${\mathrm{scriptS}}_1\cup {\mathrm{scriptS}}_{\text{WBC}}\cup {\mathrm{scriptS}}_{2*}$.…”

“…Obtain a candidate subset $\stackrel{true^}{{\mathrm{scriptS}}_1}$ of strong signals using a penalized regression method. Here, we consider the penalized least squares (PLS) estimator from Gao et al (2017): $${\stackrel{true^}{bold-italic\beta }}^{\text{PLS}}=\underset{bold-italic\beta }{\text{argmin}}\left\{\Vert y-X\beta {false\Vert }_2^2+\sum _{j=1}^p{\text{Pen}}_{\lambda }\left({\beta }_j\right)\right\},$$ where Pen λ ( β j ) is a penalty on each individual β j to shrink the weak effects toward zeros and select the strong signals, with the tuning parameter λ > 0 controlling the size of the candidate subset $\stackrel{true^}{{\mathrm{scriptS}}_1}$. Commonly used penalties are Pen λ ( β j ) = λ| β j | and Pen λ ( β j ) = λ ω j | β j | for Lasso and adaptive Lasso, where ω j > 0 is a known weight.…”

“…As correlations are ranged between −1 and 1, it is easier to set a target range for α . To detect signals in ${\mathrm{scriptS}}_{2*}$, we follow Gao et al (2017) to use cross-validation to choose ${\tilde{\lambda }}_n$ and a n in (8) and (9), respectively. In particular, we set and ${\tilde{\lambda }}_n=c_1{a}_n^{-2}{(\text{loglog}n)}^3\text{log}(n\vee p)$ a n = c 2 n −1/8 for some positive constants c 1 and c 2 .…”

“…(A8) ensures that the precision matrix can be accurately estimated by inverting the thresholded sample covariance matrix; see Shao et al (2011) and Bickel and Levina (2008) for details. (A9), which bounds the total size of weak signals on ${\mathrm{scriptS}}_{2*}$, is required for selection consistency on ${\mathrm{scriptS}}_{2*}$ (Gao et al, 2017). …”

“…We conduct simulations to compare the performance of the proposed CIS-PSE and the post-shrinkage estimator (PSE) by Gao et al (2017). The key difference between CIS-PSE and PSE lies in that PSE focuses only on ${\mathrm{scriptS}}_1$ whereas CIS-PSE considers ${\mathrm{scriptS}}_1\cup {\mathrm{scriptS}}_{\text{WBC}}$.…”