Budding Trees

Abstract

We propose a new decision tree model, named the budding tree, where a node can be both a leaf and an internal decision node. Each bud node starts as a leaf node, can then grow children, but then later on, if necessary, its children can be pruned. This contrasts with traditional tree construction algorithms that only grows the tree during the training phase, and prunes it in a separate pruning phase. We use a soft tree architecture and show that the tree and its parameters can be trained using gradient-descent. Our experimental results on regression, binary classification, and multi-class classification data sets indicate that our newly proposed model has better performance than traditional trees in terms of accuracy while inducing trees of comparable size.

One-Stage Tree: end-to-end tree builder and pruner

“…Hard Tree selects a subpath for instances according to a specific feature and threshold. In the multivariate tree (Irsoy et al, 2012;Norouzi et al, 2015;Irsoy et al, 2014;Hehn et al, 2019), which is also called soft tree, i is a continuous variable and s(x; i ) defines an oblique split.…”

“…Unlike Soft Decision Tree that only searches the splitting rule, Budding Tree (Irsoy et al, 2014) relaxes and fits the tree architecture. The bud node i can be an internal node and a leaf at the same time according to i .…”

“…Moreover, despite the low empirical error, decision trees are easily overfitted (Kotsiantis, 2013). To improve learning through end-to-end training with back-propagation, soft trees (Norouzi et al, 2015;Irsoy et al, 2014;Hehn et al, 2019) are proposed.…”

“…That is, is more crucial in some way. Budding Tree (Irsoy et al, 2014) considers the architecture parameters, but its randomly pruned nodes fail to bud. All of the above methods directly utilize probabilistic trees when testing, which looses the interpretability of decision trees.…”

Continuously Constructive Deep Neural Networks

“…Our second approach is inspired by the budding tree, which we proposed before, and it is a tree where complexity is softly parameterized [12]. Unlike a regular tree where a node is either a decision node or a leaf node, in a budding tree each node m has a leafness parameter γ m ∈ [0, 1] and is both a leaf node with weight γ m and an internal node with weight 1 − γ m .…”