C443: a Methodology to See a Forest for the Trees

Abstract

Often tree-based accounts of statistical learning problems yield multiple decision trees which together constitute a forest. Reasons for this include examining tree instability, improving prediction accuracy, accounting for missingness in the data, and taking into account multiple outcome variables. A key disadvantage of forests, unlike individual decision trees, is their lack of transparency. Hence, an obvious challenge is whether it is possible to recover some of the insightfulness of individual trees from a forest. In this paper, we will propose a conceptual framework and methodology to do so by reducing forests into one or a small number of summary trees, which may be used to gain insight into the central tendency as well as the heterogeneity of the forest. This is done by clustering the trees in the forest based on similarities between them. By means of simulated data we will demonstrate how and why different similarity types in the proposed methodology may lead to markedly different conclusions, and explain when and why certain approaches may be recommended over other ones. We will finally illustrate the methodology with an empirical data set on the prediction of cocaine use on the basis of personality characteristics.

SUBiNN: a stacked uni- and bivariate kNN sparse ensemble

“…RF and other classification tree ensembles, however, are often considered black-box techniques (similar to SVM). Recently, there have been advances in developing interpretational tools for RF and other black box techniques (Sies and Van Mechelen 2020;Ribeiro et al 2016). Others have adapted ensemble methods with trees to increase interpretability (Meinshausen 2010) and searched for optimal tree ensembles (Khan et al 2020(Khan et al , 2021.…”

BELLATREX: Building Explanations through a LocaLly AccuraTe Rule EXtractor

“…This representation was proposed by [17], where a variety of possibilities was presented, and it consists of an adaptation of the approach from [27], that considered a binary representation. We prefer the former representation over a binary one for its ability to take into account multiple splits on the same feature.…”

“…For this purpose we select from the related work Section 2 the methods that: 1) have a publicly available source code, and 2) provide a direct way to retrieve the predictions of the surrogate model. The selection narrows down to the C443 [17] method, which is limited to the binary classification set-up. The authors propose a similar approach to our work with their tree extraction algorithm but focus more on giving an overview of the possible approaches rather than optimising the performance of a surrogate model.…”

“…Finally, a recent work by [17] was built upon the aforementioned [10], suggesting a range of tree-distance similarity measures (tree-metrics) and performing clustering on such tree representations. The authors extracted a few trees out of a random forest and illustrate the validity of their methodology using only a single empirical dataset.…”

“…For this purpose, we define a distance on the tree-representation space. We use a generalised version of the Jaccard similarity index by [17], where the ratio between the elementwise minimum and maximum of two vectors is taken. Namely, given two tree representations w i and w j , we compute:…”