A New Principle for Tuning-Free Huber Regression

Abstract

The robustification parameter, which balances bias and robustness, has played a critical role in the construction of sub-Gaussian estimators for heavy-tailed and/or skewed data. Although it can be tuned by cross-validation in traditional practice, in large scale statistical problems such as high dimensional covariance matrix estimation and large scale multiple testing, the number of robustification parameters scales with the dimensionality so that cross-validation can be computationally prohibitive. In this paper, we propose a new data-driven principle to choose the robustification parameter for Huber-type sub-Gaussian estimators in three fundamental problems: mean estimation, linear regression, and sparse regression in high dimensions. Our proposal is guided by the non-asymptotic deviation analysis, and is conceptually different from cross-validation which relies on the mean squared error to assess the fit. Extensive numerical experiments and real data analysis further illustrate the efficacy of the proposed methods.

Large-Scale Inference of Multivariate Regression for Heavy-Tailed and Asymmetric Data

“…, p with high probability even when p grows exponentially fast with n. Here, the divergence of τ j guarantees p θ j to be sub-Gaussian even the error only admits p2 δqth finite moment, and more importantly, the order of τ j grants the desired approximation error of Bahadur representation to p θ j (Proposition 1) as well as the uniform non-asymptotic bounds of the estimated covariance of p θ j (Theorem 2). As noticed in the literature (Catoni, 2012;Fan et al, 2019;Sun et al, 2020;Wang et al, 2021), the divergent τ j is necessary to balance the bias and robustness in the presence of heavy-tailed and/or skewed errors. On the other hand, the order of τ j in our setting is different from the earlier studies on the adaptive Huber regressions.…”

“…For example, with the finite p1 qth moment of error, Sun et al (2020) focused on estimating the adaptive Huber regression that corresponds to p 1 in our setting and considered τ j Opn maxt1{p1 q,1{2u pd log nq ¡ maxt1{p1 q,1{2u q, while Fan et al ( 2019) used τ j Opn 1{2 tlogpnpqu ¡1{2 q for testing p-dimensional mean vectors under the assumption of finite fourth moment of errors, which corresponds to d 1 in our setting. In practice, τ j can be chosen by either the 2.1 Test procedure for general linear hypotheses10 cross-validation or the recent data-driven method by Wang et al (2021).…”

“…Theoretically, the non-asymptotic Bahadur representation is a crucial step for establishing the limiting distribution of the estimator or its functionals. Practically, the proposed method can be fully data-driven (Wang et al, 2021), and therefore is computationally attractive and applicable to real large-scale problems.…”

“…For that, we explore and develop a couple of interesting new technical results, by which we show that the proposed method controls the false discovery proportion and rate asymptotically under mild moment and correlation conditions on the error vector. Computationally, our method is fast by taking advantage of the computational efficiency of data-adaptive Huber regression (Wang et al, 2021). In addition to numer-ical experiments, we apply our method to analyze the text data from the Standardized Gutenberg Project Corpus (Gerlach and Font-Clos, 2018).…”

Robust and flexible inference for the covariate‐specific receiver operating characteristic curve

“…Before proceeding it is worth mentioning that although all tuning constants were set using the standard normal distribution as a benchmark, the location‐scale regression model in () only requires the error term to have zero mean and unit variance. However, if the error distribution is asymmetric, the choice of induces bias at the intercept estimate and, consequently, the corresponding prediction of the conditional mean is also biased 16,17 . Increasing the tuning constant reduces the bias but this parameter cannot be increased too much in order to maintain the robustness.…”