Parameterized Complexity of Small Decision Tree Learning

Abstract

We study the NP-hard problem of learning a decision tree (DT) of smallest depth or size from data. We provide the first parameterized complexity analysis of the problem and draw a detailed parameterized complexity map for the natural parameters: size or depth of the DT, maximum domain size of all features, and the maximum Hamming distance between any two examples. Our main result shows that learning DTs of smallest depth or size is fixed-parameter tractable (FPT) parameterized by the combination of all three of these parameters. We contrast this FPT-result by various hardness results that underline the algorithmic significance of the considered parameters.

The Influence of Dimensions on the Complexity of Computing Decision Trees

“…Thus, the total running time adds up to O(D 2d max nd) = O(n 2d+1 d). Before we elaborate on our next result, we first show how to improve Theorem 4 in (Ordyniak and Szeider 2021), which shows that, given an instance (E, λ, s) of DTS, and given a subset D of the dimensions, it is possible to compute in 2 O(s 2 ) |E| 1+o(1) log |E| time the smallest decision tree among all decision trees of size at most s (if they exists), that use exactly the given subset D in their cuts -none may be left out. The main idea is to first enumerate the structure of all possible decision trees of size s, before finding thresholds that work for the instance.…”

“…The strategy employed by Ordyniak and Szeider to solve DTS is to first find a subset D of dimensions which should be cut to find a smallest decision tree (Ordyniak and Szeider 2021). Once the set D has been determined, they use their Theorem 4 to find a smallest decision tree.…”

“…The classical CART heuristic herein is among the Top 10 Algorithms of Data Mining chosen by the ICDM (Wu et al 2008;Steinberg 2009) and several implementations are based on exact algorithms minimizing the size of the produced trees. Despite this, our knowledge of the computational complexity of learning (minimum-node) decision trees is limited: Several classical results show NPhardness (Hyafil and Rivest 1976;Goodrich et al 1995) (see also the survey by Murthy (1998)) and we know that even if we require parameters such as the number of nodes of the tree, or the number of different feature values, to be small, we still cannot achieve efficient algorithms in terms of upper bounds on the running time (Ordyniak and Szeider 2021).…”

“…In contrast, we show that in the small-dimension regime we do obtain a prospect for an efficient algorithm with running time O((s 3 d) s • n 1+o( 1) ) (Theorem 3.2). An intermediate result towards this is inspired by and improves upon an algorithm by Ordyniak and Szeider (2021) for determining a smallest decision tree that cuts only a given set of features; we decrease the running time from 1) for our purpose.…”

Learning Small Decision Trees for Data of Low Rank-Width

“…This complexity barrier motivates the study of the problem under the parameterized complexity paradigm. Ordyniak and Szeider (2021) made the first approach in this direction, parameterizing the problem by solution size in terms of the number of nodes or the depth of the computed DT. In this paper, we parameterize the problem to exploit the hidden structure of the given CI E. We capture the hidden structure of E in terms of small rank-width of the incidence graph, which is the bipartite graph G(E) whose vertices are the examples in one part and the features in the other, where an example e is adjacent to a feature f if and only if e(f ) = 1.…”