Learning optimal decision trees using constraint programming

Abstract

Decision trees are among the most popular classification models in machine learning. Traditionally, they are learned using greedy algorithms. However, such algorithms have their disadvantages: it is difficult to limit the size of the decision trees while maintaining a good classification accuracy, and it is hard to impose additional constraints on the models that are learned. For these reasons, there has been a recent interest in exact and flexible algorithms for learning decision trees. In this paper, we introduce a new approach to learn decision trees using constraint programming. Compared to earlier approaches, we show that our approach obtains better performance, while still being sufficiently flexible to allow for the inclusion of constraints. Our approach builds on three key building blocks: (1) the use of AND/OR search, (2) the use of caching, (3) the use of the CoverSize global constraint proposed recently for the problem of itemset mining. This allows our constraint programming approach to deal in a much more efficient way with the decompositions in the learning problem. * This paper is an extended abstract of Learning Optimal Decision Trees using Constraint Programming presented at The 25th International Conference on Principles and Practice of Constraint Programming (CP2019) [Verhaeghe et al., 2019].

Machine Learning in Agriculture: A Comprehensive Updated Review

“…In particular, each tree’s node denotes a dissimilar pairwise comparison regarding a certain feature, while each branch corresponds to the result of this comparison. As regards leaf nodes, they stand for the final decision/prediction provided after following a certain rule [ 91 , 92 ]. As for Regression, it is used for supervised learning models intending to model a target value on the basis of independent predictors.…”

Mathematical optimization in classification and regression trees

“…In this section, we focus on the Continuous Optimization and the Mixed-Integer Linear Optimization paradigms (Bertsimas and Dunn 2017;Blanquero et al 2020b;Firat et al 2020;Günlük et al 2019). The reader is referred to, e.g., Verhaeghe et al (2019) for a constraint programming paradigm, an SAT one in Narodytska et al (2018), Yu et al (2020), and a dynamic programming one in . This section aims at comparing the two paradigms in terms of type of decision variables and constraints required to model (1) the movement of individuals along the tree and (2) the prediction rule for new individuals.…”

Learning Optimal Decision Trees Using Caching Branch-and-Bound Search

“…Aghaei, Azizi, and Vayanos (2019) exploit this to formalize a learning problem that also takes into account the fairness of a prediction. Verhaeghe et al (2019) recently proposed a Constraint Programming (CP) approach to solve the same problem. It supports a maximum depth constraint and a minimum support constraint, but only works for binary classification tasks.…”