An Optimal Constrained Pruning Strategy for Decision Trees

Abstract

T his paper is concerned with the optimal constrained pruning of decision trees. We present a novel 0-1 programming model for pruning the tree to minimize some general penalty function based on the resulting leaf nodes, and show that this model possesses a totally unimodular structure that enables it to be solved as a shortest-path problem on an acyclic graph. Moreover, we prove that this problem can be solved in strongly polynomial time while incorporating an additional constraint on the number of residual leaf nodes. Furthermore, the framework of the proposed modeling approach renders it suitable to accommodate different (multiple) objective functions and side-constraints, and we identify various such modeling options that can be applied in practice. The developed methodology is illustrated using a numerical example to provide insights, and some computational results are presented to demonstrate the efficacy of solving generically constrained problems of this type. We also apply this technique to a large-scale transportation analysis and simulation system (TRANSIMS), and present related computational results using real data to exhibit the flexibility and effectiveness of the proposed approach.

Optimally Pruning Decision Tree Ensembles With Feature Cost

“…Proposition 1 of (Sherali et al, 2009) showed that the following set of constraints completely characterize the set of valid pruned trees of T .…”

“…By showing that the constraint matrix can be turned into a network matrix form, (Sherali et al, 2009) showed the above integer problem can be solved exactly by linear program relaxation.…”

Mathematical optimization in classification and regression trees

“…This network design problem is often avoided, by, e.g., choosing a binary tree of depth D, for a given value of D. To make this decision more data-dependent, a larger tree is built and pruned afterward, collapsing existing leaf nodes into new ones containing more individuals. See, e.g., Sherali et al (2009) for structural properties of the optimization problem associated with the pruning step. In this way, one obtains a more parsimonious tree, which is expected to perform better for future individuals.…”

“…See, e.g., Sherali et al. ( 2009 ) for structural properties of the optimization problem associated with the pruning step. In this way, one obtains a more parsimonious tree, which is expected to perform better for future individuals.…”

A noise handling method for hyper surface classification

“…As a common framework for avoiding the problem of overfitting noisy data, pruning algorithms have been proposed for many classification algorithms. In decision tree algorithms [2] [3], there are many classical pruning methods proposed such as Reduced Error Pruning (REP) [4], Pessimistic Error Pruning (PEP) [4], Minimum Error Pruning (MEP) [5], CostComplexity Pruning (CCP) [2] and new pruning methods like k-norm pruning [6], optimal constrained pruning [7] are being proposed continually. For separate-and-conquer rulelearning systems, pruning techniques are also important for noise handling.…”