The random subspace method for constructing decision forests

Abstract

Much of previous attention on decision trees focuses on the splitting criteria and optimization of tree sizes. The dilemma between overfitting and achieving maximum accuracy is seldom resolved. A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity. The classifier consists of multiple trees constructed systematically by pseudorandomly selecting subsets of components of the feature vector, that is, trees constructed in randomly chosen subspaces. The subspace method is compared to single-tree classifiers and other forest construction methods by experiments on publicly available datasets, where the method's superiority is demonstrated. We also discuss independence between trees in a forest and relate that to the combined classification accuracy.

An improved approach for prediction of Parkinson's disease using machine learning techniques

“…In Random forest, there are many decision trees. For a given input, each of the decision trees classify it as yes/no (in case of binary classification) [24] [25]. Then once each of the trees have classified as yes/no, the value which has the majority among them is taken as output.…”

Random Subspace Ensemble Artificial Neural Networks for First-episode Schizophrenia Classification

“…To ensure a necessary assumption, which is the disparity among the classifiers, variability in the training process must be somehow acquired. Many methods have been invented for this purpose, such as random subspace ensemble [13], random forests [14], bagging [15], boosting (e.g. AdaBoost [16]), rotation forests [17] and others.…”

Detecting Unknown Malware on Android by Machine Learning Using the Feature of Dalvik Operation Code

“…[15] The extension combines Breiman's "bagging" idea and random selection of features, introduced first by Ho [1] and later independently by Amit and Geman [16] in order to construct a collection of decision trees with controlled variance.…”

“…As a result of this randomness, the bias of the forest usually slightly increases (with respect to the bias of a single nonrandom tree) but, due to averaging, its variance also decreases, usually more than compensating for the increase in bias, hence yielding an overall better mode [1] [18].…”

“…The reason for doing this is the correlation of the trees in an ordinary bootstrap sample: if one or a few features are very strong predictors for the response variable (target output), these features will be selected in many of the B trees, causing them to become correlated. An analysis of how bagging and random subspace projection contributes to accuracy gains under different conditions is given by Ho [1].…”