Improving Deep Forest by Exploiting High-order Interactions

Chen, Yi-He; Lyu, Shen-Huan; Jiang, Yuan

doi:10.1109/icdm51629.2021.00118

Cited by 7 publications

(16 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Various studies show that considering interactions between features of training data helps improve accuracy of decision trees and their ensembles [13]. Training trees with oblique (linear) or more complex decision splits is a natural way to consider such interactions [14].…”

Section: B Oblique Decision Trees and Random Forestsmentioning

confidence: 99%

Random Kernel Forests

Devyatkin

Grigoriev

2022

IEEE Access

View full text Add to dashboard Cite

Random forests of axis-parallel decision trees still show competitive accuracy in various tasks; however, they have drawbacks that limit their applicability. Namely, they perform poorly for multidimensional sparse data that is frequently input in many text and image processing tasks. A straightforward solution is to create forests of decision trees with oblique splits; however, most training approaches have low performance or are hard to tune. Besides, those ensembles appear unstable and easily overfit, so they should be regularized to find the trade-off between complexity and generalization. This paper proposes an algorithm to train kernel decision trees and random forests. At the top level, this algorithm follows a common greedy procedure to train binary decision trees; however, it produces quasi-optimal linear and kernel splits at the decision stump level. At each stump, the algorithm finds a quasi-optimal distribution of classes to subtrees and trains this stump via optimization of an SVM-like loss function with a margin re-scaling approach, which helps simultaneously optimize the margin between subtree data and arbitrary impurity criteria. We also try to reveal uniform stability-based generalization bounds for those forests and apply them to select the regularization technique. The obtained bounds explicitly consider primary hyperparameters of forests, trees, and decision stumps. The bounds also show a relationship between outliers in decision tree structure, stability and generalization, and how forests smooth these outliers. We performed an experimental evaluation on several tasks, such as studying the reaction of social media users, image recognition, and bank scoring. The experimental results show that the proposed algorithm trains ensembles, which outperform other oblique or kernel forests in many commonly-recognized datasets. Namely, the proposed algorithm shows 99.1% accuracy on MNIST and 58.1% on CIFAR-10. It has been confirmed that the selected regularization technique helps reduce overfitting on several datasets. Therefore, the proposed algorithm may be considered a small step toward customized and regularized ensembles of oblique or kernel trees that keep reasonable training performance on large datasets. We believe that the proposed algorithm can be utilized as a construction element of approaches to training context-aware models on complex problems with integer inputs and outputs, such as text mining and image recognition.INDEX TERMS Generalization bounds, kernel splits, kernel forests, random forests, random uniform stability, regularization, slack re-scaling.

show abstract

Section: B Oblique Decision Trees and Random Forestsmentioning

confidence: 99%

Random Kernel Forests

Devyatkin

Grigoriev

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

“…For example, Chen et al argue that the prediction-based feature representation of Deep Forest is a critical deficiency because the predicted class probabilities deliver very limited information [4]. They present a deep forest model that utilizes high-order interactions of input features to generate more informative and diverse feature representations [4]. They created a generalized version of Random Intersection Trees to reveal stable high-order relationships and apply activated linear combinations to transform them into hierarchical distributed representations.…”

Section: Related Workmentioning

confidence: 99%

“…Deep forest uses a cascade structure to perform layer-by-layer processing of raw features. However, several studies report that Deep Forest has limited ability to catch dependencies between features, which can lead to poor performance in some cases [4]. Besides, it uses Extremely Random Forests to control overfitting, which in practice can lead to unstable results.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Estimation of Vegetation Indices with Random Kernel Forests

Devyatkin¹

2022

Preprint

View full text Add to dashboard Cite

Vegetation indexes help perform precision farming because they provide useful information regarding moisture, nutrient content, and crop health. Primary sources of those indexes are satellites and unmanned aerial vehicles equipped with expensive multispectral sensors. Reducing the price of obtaining such information would increase the availability of precision farming for small farms. Several studies have proposed deep neural network methods to estimate the indexes from RGB color images. However, these methods report relatively large errors for mature plants when highly non-linear relationships of images and vegetation indexes arise. One could apply multilayer random forest-based models (Deep Forests) to solve this problem, but the discriminative power of such models is limited: they cannot catch complex dependencies between image features. In this paper, we propose a method that combines ideas of deep forests, random forests of kernel trees, and global pruning of random forests to tackle the problem. As a result, the method considers the properties of objects with a complex structure: the presence of relationships between groups of features, displacement, and scaling of objects. The experimental results show that the proposed method outperforms neural network-based solutions in several datasets.

show abstract

“…Chen et al . [ 31 ] design an interaction‐based deep forest to improve testing efficiency and enrich new features.…”

Section: Introductionmentioning

confidence: 99%

A Region‐Based Analysis for the Feature Concatenation in Deep Forests

Lyu

CHEN

Zhou

2022

Chinese J of Electronics

View full text Add to dashboard Cite

Deep forest is a tree‐based deep model made up of non‐differentiable modules that are trained without backpropagation. Despite the fact that deep forests have achieved considerable success in a variety of tasks, feature concatenation, as the ingredient for forest representation learning, still lacks theoretical understanding. In this paper, we aim to understand the influence of feature concatenation on predictive performance. To enable such theoretical studies, we present the first mathematical formula of feature concatenation based on the two‐stage structure, which regards the splits along new features and raw features as a region selector and a region classifier respectively. Furthermore, we prove a region‐based generalization bound for feature concatenation, which reveals the trade‐off between Rademacher complexities of the two‐stage structure and the fraction of instances that are correctly classified in the selected region. As a consequence, we show that compared with the prediction‐based feature concatenation (PFC), the advantage of interaction‐based feature concatenation (IFC) is that it obtains more abundant regions through distributed representation and alleviates the overfitting risk in local regions. Experiments confirm the correctness of our theoretical results.

show abstract

Improving Deep Forest by Exploiting High-order Interactions

Cited by 7 publications

References 16 publications

Random Kernel Forests

Random Kernel Forests

Estimation of Vegetation Indices with Random Kernel Forests

A Region‐Based Analysis for the Feature Concatenation in Deep Forests

Contact Info

Product

Resources

About