Simultaneous Feature Selection and Outlier Detection with Optimality Guarantees

Abstract

Sparse estimation methods capable of tolerating outliers have been broadly investigated in the last decade. We contribute to this research considering high-dimensional regression problems contaminated by multiple mean-shift outliers which affect both the response and the design matrix. We develop a general framework for this class of problems and propose the use of mixed-integer programming to simultaneously perform feature selection and outlier detection with provably optimal guarantees. We characterize the theoretical properties of our approach, i.e. a necessary and sufficient condition for the robustly strong oracle property, which allows the number of features to exponentially increase with the sample size; the optimal estimation of the parameters; and the breakdown point of the resulting estimates. Moreover, we provide computationally efficient procedures to tune integer constraints and to warm-start the algorithm. We show the superior performance of our proposal compared to existing heuristic methods through numerical simulations and an application investigating the relationships between the human microbiome and childhood obesity.

Doubly Robust Feature Selection with Mean and Variance Outlier Detection and Oracle Properties

“…However, since we rely on nonconcave penalization methods, our proposal satisfies oracle properties under weaker assumptions compared to existing robust estimators based on convex penalties (Kurnaz et al 2017;Alfons et al 2013). This provides an important bridge between the latter and L 0 -constrained formulations with optimality guarantees (Insolia et al 2020). Moreover, unlike "soft" trimming estimators which produce a general down-weighting for all points (Loh 2017;Smucler and Yohai 2017;Chang et al 2018;Freue et al 2019;Amato et al 2021), our proposal is effective in estimating full weights for non-outlying observations.…”

“…The MSOM assumes that outlying cases have a shift in mean; maximum likelihood estimation (MLE) leads to their removal from the fit -i.e., to the assignment of 0 weights to the cases identified as outliers. While the MSOM was traditionally studied in low-dimensional scenarios , it has been recently extended to high-dimensional linear models, where the use of regularization techniques is fundamental (She and Owen 2011;Alfons et al 2013;Kurnaz et al 2017;Insolia et al 2020). The VIOM, which is historically considered as an alternative to the MSOM, assumes that contaminated errors have an inflated variance; outliers are retained but down-weighted in the fit.…”

Robust Variable Selection and Estimation Via Adaptive Elastic Net S-Estimators for Linear Regression

“…Under a very general framework allowing for different loss and penalty functions, She et al (2021) establish minimax bounds for the estimation error under the MSOM and propose an efficient algorithm for estimation. In a similar sprit, Insolia et al (2020) propose a mixed integer program (MIP) to constrain both the number of outliers and the number of relevant predictors using the L 0 -pseudo-norm. The authors develop guarantees for the algorithmic complexity and the statistical estimation error for this MIP under the MSOM, even for ultra-high dimensional problems.…”

“…For lower signal strengths, however, the L 0 -pseudo-norm for either variable selection or outlier detection tends to suffer from high variability. For variable selection and prediction, for example, other penalties may deliver better performance (Hastie et al 2020;Insolia et al 2020). In addition, for computational tractability and stability, it is necessary to restrict the optimization to a tight neighborhood around the true regression parameters, as well as to have a good understanding of the number of relevant predictors and the number of outliers.…”

“…In addition, for computational tractability and stability, it is necessary to restrict the optimization to a tight neighborhood around the true regression parameters, as well as to have a good understanding of the number of relevant predictors and the number of outliers. Insolia et al (2020), for example, suggest to use an ensemble of other robust regression estimates to get preliminary estimates of the necessary bounds. In this paper we are proposing a method which does not require any prior knowledge of the number of truly relevant predictors and only a rough upper bound on the number of outliers.…”