A New Adaptive Weighted Deep Forest and Its Modifications

Abstract

A new adaptive weighted deep forest algorithm which can be viewed as a modification of the confidence screening mechanism is proposed. The main idea underlying the algorithm is based on adaptive weigting of every training instance at each cascade level of the deep forest. The confidence screening mechanism for the deep forest proposed by Pang et al., strictly removes instances from training and testing processes to simplify the whole algorithm in accordance with the obtained random forest class probability distributions. This strict removal may lead to a very small number of training instances at the next levels of the deep forest cascade. The presented modification is more flexible and assigns weights to instances in order to differentiate their use in building decision trees at every level of the deep forest cascade. It overcomes the main disadvantage of the confidence screening mechanism. The proposed modification is similar to the AdaBoost algorithm to some extent. Numerical experiments illustrate the outperformance of the proposed modification in comparison with the original deep forest. It is also illustrated how the proposed algorithm can be extended for solving the transfer learning and distance metric learning problems.

Attention-based Random Forest and Contamination Model

“…The case opt = 1 means that weights of trees are totally determined by the tree results and do not depend on each instance. This case coincides with the weighted RF proposed in [30]. The case opt = 0 means that weights of trees are determined only by the softmax function (with or without trainable parameters).…”

“…However, the assigned weights in the aforementioned works are not trainable parameters. Attempts to train weights of trees were carried out in [30,31,7,8], where weights are assigned by solving optimization problems, i.e., they incorporated into a certain loss function of the whole RF such that the loss function is minimized over values of weights.…”

“…It is well-known that this comparison of the probability distributions is not the best way in classification. However, different distances between probability distributions, for example, the Kullback-Leibler divergence, complicates the optimization problem (30), which becomes non-quadratic. Therefore, the Euclidean distance is used to save properties of the ABRF-1 simplicity.…”

DBC-Forest: Deep forest with binning confidence screening

“…AWDF: AWDF [20] is an improved deep forest method, which adopts the adaptive weight of every training instance at each cascade level.…”

“…Yang et al [19] proposed a multi-label learning deep forest technique, which employed measure-aware feature reuse and layer growth to solve a multi-label learning problem. Utkin et al [20] proposed a novel model called adaptive weighted deep forest, in which each training instance was given a weight at each cascade level of the model. Recently, gcForest has been applied in the classification of schizophrenia data [21], disease classification [22] and cancer detection [23,24,25].…”