Comparison of the Representational Power of Random Forests, Binary Decision Diagrams, and Neural Networks

Abstract

In this letter, we compare the representational power of random forests, binary decision diagrams (BDDs), and neural networks in terms of the number of nodes. We assume that an axis-aligned function on a single variable is assigned to each edge in random forests and BDDs, and the activation functions of neural networks are sigmoid, rectified linear unit, or similar functions. Based on existing studies, we show that for any random forest, there exists an equivalent depth-3 neural network with a linear number of nodes. We also show that for any BDD with balanced width, there exists an equivalent shallow depth neural network with a polynomial number of nodes. These results suggest that even shallow neural networks have the same or higher representation power than deep random forests and deep BDDs. We also show that in some cases, an exponential number of nodes are required to express a given random forest by a random forest with a much fewer number of trees, which suggests that many trees are required for random forests to represent some specific knowledge efficiently.

On the Generative Power of ReLU Network for Generating Similar Strings

“…(23) and n nodes y i , 1 ≤ i ≤ n. The five hidden layers perform the computation of Eqs. (21) and (22). The second layer keeps a copy of 2d nodes x j and performs the computation of δ(x j , x k ) with 4d 2 nodes ψ 1 kj , ψ 2 kj , ρ 1 kj , ρ 2 kj , 1 ≤ j, k ≤ d based on Prop.…”

“…Bengio et al [10] and Biau et al [11] proved that decision trees and random forests can be efficiently expressed as neural networks by using sigmoidal functions, Heaviside functions, and hyperbolic tangent functions as activation functions. Later on, Kumano and Akutsu [22], extended the result for the neural networks with rectified linear unit (ReLU) and other related activation functions.…”

“…If the function family is too extensive, it can lead to issues like high computational costs and overfitting. On the other hand, if the function family is too limited and doesn't encompass the target functions, it may not yield the desired prediction results [22]. However, selecting an appropriate network is a challenging task, and it is still unclear which network should be used for a given problem setting.…”

A Survival Status Classification Model for Osteosarcoma Patients Based on E-CNN-SVM and Multisource Data Fusion

“…The osteosarcoma gene sequencing data has a small amount of data, but its dimensionality is relatively high. The output data after initial feature selection by random forest [ 20 ], followed by deeper feature extraction using a pretrained E-CNN model, satisfies the characteristics of small data volume and low dimensionality described in the previous section. Therefore, this paper uses SVM to replace the Sigmoid activation function of the convolutional neural network as the final classifier, which can further improve the generalization ability and stability of the model.…”

“…e output data after initial feature selection by random forest [20], followed by deeper feature extraction using a pretrained E-CNN model, satisfies the characteristics of small data volume and low dimensionality described in the previous section.…”