Sparse classification: a scalable discrete optimization perspective

Abstract

We formulate the sparse classification problem of n samples with p features as a binary convex optimization problem and propose a outer-approximation algorithm to solve it exactly. For sparse logistic regression and sparse SVM, our algorithm finds optimal solutions for n and p in the 10,000 s within minutes. On synthetic data our algorithm achieves perfect support recovery in the large sample regime. Namely, there exists an n 0 such that the algorithm takes a long time to find an optimal solution and does not recover the correct support for n < n 0 , while for n ⩾ n 0 , the algorithm quickly detects all the true features, and does not return any false features. In contrast, while Lasso accurately detects all the true features, it persistently returns incorrect features, even as the number of observations increases. Consequently, on numerous real-world experiments, our outer-approximation algorithms returns sparser classifiers while achieving similar predictive accuracy as Lasso. To support our observations, we analyze conditions on the sample size needed to ensure full support recovery in classification. For k-sparse classification, and under some assumptions on the data generating process, we prove that information-theoretic limitations impose n 0 < C 2 + 𝜎 2 k log(p − k) , for some constant C > 0.

Sparse Regression: Scalable algorithms and empirical performance

“…Therefore, new research in numerical algorithms for solving the exact formulation (2) directly has flourished. Leveraging recent advances in mixed-integer solvers [6,4], Lagrangian relaxation [39] or cutting-plane methods [5,7], these works have demonstrated significant improvement over existing Lasso-based heuristics. To the best of our knowledge, the exact algorithm proposed by Bertsimas and Van Parys [5], Bertsimas et al [7] is the most scalable method providing provably optimal solutions to the optimization problem (2), at the expense of potentially significant computational time and the use of a commercial integer optimization solver.…”

“…Thanks to both hardware improvement and advances in mixed-integer optimization solvers, Bertsimas et al [6], Bertsimas and King [4] successfully used discrete optimization techniques to solve instances with n, p in the 1, 000s within minutes. More recently, Bertsimas and Van Parys [5], Bertsimas et al [7] proposed a cutting plane approach which scales to data sizes of with n, p in the 100, 000s for ordinary least square and n, p in the 10, 000s for logistic regression. To the best of our knowledge, our approach is the only method which scales to instances of such sizes, while provably solving such an NP-hard problem.…”

“…Bertsimas and Van Parys [5], Bertsimas et al [7] consider an 2 -regularized version of the initial formulation (2),…”

“…by iteratively tightening a piece-wise linear lower approximation of c. Pseudo-code is given in Algoritm 2.1 (p. 5). Proof of termination and details on implementation can be found in Bertsimas and Van Parys [5] for regression and Bertsimas et al [7] for classification. This outer-approximation scheme was originally proposed by Duran and Grossmann [15] for general nonlinear mixed-integer optimization problems.…”

“…In this section, we compare the aforementioned methods on synthetic linear regression data where the ground truth is known to be sparse. The convex integer optimization algorithm of Bertsimas and Van Parys [5], Bertsimas et al [7] (CIO in short) was implemented in Julia [34] using the commercial solver Gurobi 7.5.0 [26], interfaced using the optimization package JuMP [14]. The Lagrangian relaxation is also implemented in Julia and openly available as the SubsetSelection package (SS in short).…”