A Tuning-free Robust and Efficient Approach to High-dimensional Regression

Abstract

We introduce a novel approach for high-dimensional regression with theoretical guarantees. The new procedure overcomes the challenge of tuning parameter selection of Lasso and possesses several appealing properties. It uses an easily simulated tuning parameter that automatically adapts to both the unknown random error distribution and the correlation structure of the design matrix. It is robust with substantial efficiency gain for heavy-tailed random errors while maintaining high efficiency for normal random errors. Comparing with other alternative robust regression procedures, it also enjoys the property of being equivariant when the response variable undergoes a scale transformation. Computationally, it can be efficiently solved via linear programming. Theoretically, under weak conditions on the random error distribution, we establish a finite-sample error bound with a near-oracle rate for the new estimator with the simulated tuning parameter. Our results make useful contributions to mending the gap between the practice and theory of Lasso and its variants. We also prove that further improvement in efficiency can be achieved by a second-stage enhancement with some light tuning. Our simulation results demonstrate that the proposed methods often outperform cross-validated Lasso in various settings. Supplementary materials for this article are available online.

General Bayesian Loss Function Selection and the use of Improper Models

“…We note that one can add more losses into our framework, for example those in Black and Rangarajan (1996); Basu et al (1998) or Wang et al (2020). Also, our framework is not limited to linear regression.…”

Robust penalized extreme learning machine regression with applications in wind speed forecasting

“…where pðÁÞ is the penalty function, such as ridge penalty and lasso penalty. Some typical cases of the general formulate can be detailed in [51][52][53].…”

A proximal-proximal majorization-minimization algorithm for nonconvex tuning-free robust regression problems

“…For simplicity, we call the problem (1) the tuning-free robust Lasso problem. It has been shown in [40] that the model ( 1) is very close to the Lasso for normal random errors and is robust with substantial efficiency gain for heavy-tailed errors.…”

“…Another obstacle is that the Gaussian or sub-Gaussian error assumption can hardly be satisfied for high-dimensional microarray data, climate data, insurance claim data, e-commerce data and many other applications due to the heavy-tailed errors, which can affect the choice of λ and result in misleading results if we apply the standard procedures directly. Therefore, Wang et al [40] studied the following ℓ 1 regularized tuning-free robust regression model min…”