Simultaneous feature selection and outlier detection with optimality guarantees

Abstract

Biomedical research is increasingly data rich, with studies comprising ever growing numbers of features. The larger a study, the higher the likelihood that a substantial portion of the features may be redundant and/or contain contamination (outlying values). This poses serious challenges, which are exacerbated in cases where the sample sizes are relatively small. Effective and efficient approaches to perform sparse estimation in the presence of outliers are critical for these studies, and have received considerable attention in the last decade. We contribute to this area considering high‐dimensional regressions contaminated by multiple mean‐shift outliers affecting both the response and the design matrix. We develop a general framework and use mixed‐integer programming to simultaneously perform feature selection and outlier detection with provably optimal guarantees. We prove theoretical properties for our approach, that is, a necessary and sufficient condition for the robustly strong oracle property, where the number of features can increase exponentially with the sample size; the optimal estimation of parameters; and the breakdown point of the resulting estimates. Moreover, we provide computationally efficient procedures to tune integer constraints and warm‐start the algorithm. We show the superior performance of our proposal compared to existing heuristic methods through simulations and use it to study the relationships between childhood obesity and the human microbiome.

Robust Variable Selection with Optimality Guarantees for High-Dimensional Logistic Regression

“…Nevertheless, nowadays it can be solved effectively and at times also efficiently with specialized solvers. Importantly, it relates to the use of a trimmed loss function as in [10], and it extends the work in [7] for sparse linear regression models affected by data contamination in the form of mean-shift outliers. However, here the use of a nonlinear and nonquadratic objective function complicates the matter and requires special attention.…”

“…For instance, an ensemble method based on existing heuristic and robust procedures to create suitable big-M bounds was considered in [7]. However, a similar approach is challenging in this framework given a "pool" of openly available robust algorithms is not available for logistic regression models-unlike in linear regression.…”

“…For instance, one might consider robust counterparts of information criteria or cross-validation. In our simulation study, we do not include the L 2 -constraint and, for a given trimming level k n , we use a robust version of the Bayesian information criterion (BIC) similarly to [7]. In symbols, this is BIC = k p ln (n − k n ) + ∑ n i=1 d(x T i β, y i ), where d(x T i β, y i ) are the final deviances for a given estimator-recall that deviances corresponding to trimmed points are equal to 0.…”

“…This motivates the development of robust estimation techniques. Notably, penalized estimation and robustness with respect to the presence of outliers are very closely related topics [6][7][8], and they have recently also been combined for logistic regression settings [9,10].…”

“…Specifically, we consider an L 0 sparsity assumption on the coefficients [13] and a logistic slippage model for the outlying observations [14]. We further build upon the work in [7] and rely on L 0 -constraints to detect outlying cases and select relevant features. This requires us to solve a double combinatorial problem, across both the units and the covariates.…”