Tree ensembles for predicting structured outputs

Abstract

In this article, we address the task of learning models for predicting structured outputs. We consider both global and local prediction of structured outputs, the former based on a single model that predicts the entire output structure and the latter based on a collection of models, each predicting a component of the output structure. We use ensemble methods and apply them in the context of predicting structured outputs. We propose to build ensemble models consisting of predictive clustering trees, which generalize classification trees: these have been used for predicting different types of structured outputs, both locally and globally. More specifically, we develop methods for learning two types of ensembles (bagging and random forests) of predictive clustering trees for global and local prediction of different types of structured outputs. The types of outputs considered correspond to different predictive modelling tasks: multi-target regression, multi-target classification, and hierarchical multi-label classification. Each of the combinations can be applied both in the context of global prediction (producing a single ensemble) or local prediction (producing a collection of ensembles). We conduct an extensive experimental evaluation across a range of benchmark datasets for each of the three types of structured outputs. We compare ensembles for global and local prediction, as well as single trees for global prediction and tree collections for local prediction, both in terms of predictive performance and in terms of efficiency (running times and model complexity). The results show that both global and local tree ensembles perform better than the single model counterparts in terms of predictive power. Global and local tree ensembles perform equally well, with global ensembles being more efficient and producing smaller models, as well as needing fewer trees in the ensemble to achieve the maximal performance.

Detecting flooded areas with machine learning techniques: case study of the Selška Sora river flash flood in September 2007

“…To extract flooded areas, we tested different methods involving two machine learning software packages: Weka 12 and Clus. 13 …”

Comments on: A random forest guided tour

“…Blockeel et al 1998;Geurts et al 2006b;Segal and Xiao 2011;Kocev et al 2013). These extensions are obtained by changing the CART-split criterion and by attaching vectorial predictions at tree leaves.…”

Ontology of core data mining entities

“…The primitive output prediction tasks include classification and regression, while the structured output prediction tasks include multi-target prediction (multi-target classification and multi-target regression) (Struyf and Dzeroski 2005), hierarchical classification (Vens et al 2008), multi-label classification (Madjarov et al 2012), and time-series prediction (Slavkov et al 2010). Finally, the Clus system was extended with ensemble learning algorithms for all these tasks (Kocev et al 2013). …”

“…Finally, in order to reason with the ontology and pose queries, we populated OntoDM-core with 40 classes and 79 instances of datasets that were used in real experiments with the algorithms from the Clus system published in several publications (Kocev et al 2013;Madjarov et al 2012). Each dataset instance is specified with a dataset specification, containing the specification of the descriptive and output (or target) datatypes of the data examples in the dataset.…”