Strong Optimal Classification Trees

Abstract

Decision trees are among the most popular machine learning models and are used routinely in applications ranging from revenue management and medicine to bioinformatics. In this paper, we consider the problem of learning optimal binary classification trees. Literature on the topic has burgeoned in recent years, motivated both by the empirical suboptimality of heuristic approaches and the tremendous improvements in mixedinteger optimization (MIO) technology. Yet, existing MIO-based approaches from the literature do not leverage the power of MIO to its full extent: they rely on weak formulations, resulting in slow convergence and large optimality gaps. To fill this gap in the literature, we propose an intuitive flow-based MIO formulation for learning optimal binary classification trees. Our formulation can accommodate side constraints to enable the design of interpretable and fair decision trees. Moreover, we show that our formulation has a stronger linear optimization relaxation than existing methods. We exploit the decomposable structure of our formulation and max-flow/min-cut duality to derive a Benders' decomposition method to speed-up computation. We propose a tailored procedure for solving each decomposed subproblem that provably generates facets of the feasible set of the MIO as constraints to add to the main problem. We conduct extensive computational experiments on standard benchmark datasets on which we show that our proposed approaches are 31 times faster than state-of-the art MIO-based techniques and improve out of sample performance by up to 8%.

Interpretable data‐driven contingency classification for real‐time corrective security‐constrained economic dispatch

“…Building upon SOCTs [26,27], the decision tree problem is formulated in a max-flow based model for computational efficiency.…”

“…The SOCT model [26,27], especially its branching constraints, is constructed for binary features. These constraints are as follows:…”

“…Constraints (24) indicate that data sample j can be counted as at most one in the data flow. Constraints (25) ensure that if data sample j arrives at a branch node n, then it must go to its leftor right-child node, or the sink node S. Constraints (26) ensure that if data sample j arrives at a terminal node n, then it must go to the sink node S. Constraints (27) require that only when data sample j is correctly classified to the category at node n under contingency c, then it can flow into the sink node S through node n. Constraints (28) ensure that there are at least N min data samples reaching each leaf node n.…”

“…with a loose linear programming relaxation and thus time-consuming to prove the optimality). To avoid the use of big-M constraints for computational efficiency, strong optimal classification trees (SOCTs) were developed [26,27] to transform the OCT problem into a max-flow based model from a source node to a sink node. However, SOCT only considered binary input features, while features of the corrective SCED problem, such as net load values, are continuous.…”

Fast Sparse Decision Tree Optimization via Reference Ensembles

“…Recently, many works have directly optimized a performance metric (e.g., accuracy) with soft or hard sparsity constraints on the tree size. Such decision tree optimization problems can be formulated using mixed integer programming (MIP) (Bertsimas and Dunn 2017;Verwer and Zhang 2019;Vilas Boas et al 2021;Günlük et al 2021;Rudin and Ertekin 2018;Aghaei, Gómez, and Vayanos 2021). Other approaches use SAT solvers to find optimal decision trees (Narodytska et al 2018;Hu et al 2020), though these techniques require data to be perfectly separable, which is not typical for machine learning.…”