A Dynamic Programming Based Pruning Method for Decision Trees

Abstract

This paper concerns a decision-tree pruning method, a key issue in the development of decision trees. We propose a new method that applies the classical optimization technique, dynamic programming, to a decision-tree pruning procedure. We show that the proposed method generates a sequence of pruned trees that are optimal with respect to tree size. The dynamic-programming-based pruning (DPP) algorithm is then compared with cost-complexity pruning (CCP) in an experimental study. The results of our study indicate that DPP performs better than CCP in terms of classification accuracy.

An Optimal Constrained Pruning Strategy for Decision Trees

“…Hence, the CCP method does not determine optimal solutions for each designated number of leaf nodes. Moreover, in the computational results presented in Li et al (2001), the DPP algorithm is shown to perform better than the CCP procedure in terms of classification accuracy. It is also worth noting that the sequence of pruned trees produced by the DPP approach are not necessarily nested, nor are they unique, but as Li et al (2001) argue, this issue is not of concern in regard to the quality of these trees.…”

“…Likewise, when minimizing the error measure defined by (5), we would like to force the tree to be pruned to a specified maximum size imposed by a side-constraint of the type h∈ z h ≤ E max . Observe that the DPP procedure discussed in §1 determines optimal solutions to the particular Problem PP1 with the additional constraint h∈ z h = E max simultaneously for all possible values of E max , and the CCP method solves this problem only for some subset of such values (see Li et al 2001 and the illustrative example given in §4). On the other hand, our modeling approach directly determines an optimal solution for any of the foregoing constraints having specified particular parameter values.…”

“…Some additional objective criteria, as motivated by our transportation planning case study, are discussed in detail in §3, along with related accompanying side-constraints. Li et al (2001) propose an interesting alternative dynamic-programming-based pruning (DPP) algorithm. For each possible tree size defined in terms of the number of leaf nodes, this routine recursively determines a corresponding optimal pruned tree that would have a minimal prediction error.…”

“…Lin et al (1992) have subsequently analyzed the complexity of solving such problems. Using a dynamic programming framework similar to that of Li et al (2001), they establish that when the single constraint simply restricts the number of final leaf nodes generated, then the underlying problem can be solved in polynomial time. On the other hand, under more intricate single constraints that limit the entropy or the expected depth of the leaf nodes, they demonstrate that the pruning problem becomes NP-hard.…”

“…The problem is formulated as a parametric bilinear program, which is NP-hard, and a heuristic based on the Frank-Wolfe method is adopted to solve this problem. For further reading and surveys on constructing decision trees, we refer the reader to Breslow and Aha (1997), Li et al (2001), Murthy (1997Murthy ( , 1998, and Safavin and Landgrebe (1991).…”