Robust Variable Selection With Exponential Squared Loss

Abstract

Robust variable selection procedures through penalized regression have been gaining increased attention in the literature. They can be used to perform variable selection and are expected to yield robust estimates. However, to the best of our knowledge, the robustness of those penalized regression procedures has not been well characterized. In this paper, we propose a class of penalized robust regression estimators based on exponential squared loss. The motivation for this new procedure is that it enables us to characterize its robustness that has not been done for the existing procedures, while its performance is near optimal and superior to some recently developed methods. Specifically, under defined regularity conditions, our estimators are n-consistent and possess the oracle property. Importantly, we show that our estimators can achieve the highest asymptotic breakdown point of 1/2 and that their influence functions are bounded with respect to the outliers in either the response or the covariate domain. We performed simulation studies to compare our proposed method with some recent methods, using the oracle method as the benchmark. We consider common sources of influential points. Our simulation studies reveal that our proposed method performs similarly to the oracle method in terms of the model error and the positive selection rate even in the presence of influential points. In contrast, other existing procedures have a much lower non-causal selection rate. Furthermore, we re-analyze the Boston Housing Price Dataset and the Plasma Beta-Carotene Level Dataset that are commonly used examples for regression diagnostics of influential points. Our analysis unravels the discrepancies of using our robust method versus the other penalized regression method, underscoring the importance of developing and applying robust penalized regression methods.

Influence functions for penalized M-estimators

“…However, working explicitly with the Dirac delta function understood informally as δ(x) = +∞ if x = 0 0 otherwise and inverting a matrix containing such an expression, is not fully satisfactory from a formal mathematical standpoint. Interestingly, the expression obtained in Theorem 3 in Wang et al [2013] is the same as the one we give in Proposition 3. This suggest that a more careful treatment of the problem with a rigorous use of differentiation in the sense of distribution theory will yield the same influence function.…”

“…As shown in this article, in both situations, the local robustness properties are a direct result of the form and boundedness of the derivative of the loss function. Extensive simulations illustrating the impact of deviations from the stochastic assumptions on penalized M-estimators can be found, among many others, in Sardy et al [2001], Li et al [2011], Wang et al [2013], Lozano and Meinshausen [2013] and Avella-Medina and Ronchetti [2014].…”

“…As mentioned in the introduction, Wang et al [2013] studied the robustness properties of their estimator by calculating its finite sample breakdown point and influence function. It can be seen that their derivation of the influence function (Theorem 3) could easily be extended to more general loss functions.…”

“…All these procedures rely on the intuition that a loss function that defines robust estimates in the well understood unpenalized fixed dimensional M-estimator, should also define a robust estimator when it is penalized by a deterministic function. To the best of our knowledge, only Alfons et al [2013] and Wang et al [2013] established formal robustness properties for their estimators. Both papers computed finite sample breakdown points.…”

A Robust Estimation Method for the Partial Linear Models with Longitudinal Data

“…利用核估计以及差分法消去未知的基准函数 ( ) g ⋅ ，并利用经验似然 (Owen, 1988) [6]方法估计 β 以及进行 统计推断。以上方法都没有考虑纵向数据模型中的组内相关性以及估计的稳健性，而在对实际问题的研 究中得到的数据常常存在异常值或与假定的分布不符，于是估计的稳健性变得尤为重要。 本文首先利用广义估计方程 (Liang, 1986) [7]的思想，引入工作相关矩阵将组内相关性考虑进来，然后利 用指数平方损失函数 (Wang, 2013) …”