Multistrategy ensemble learning: reducing error by combining ensemble learning techniques

Abstract

Ensemble learning strategies, especially Boosting and Bagging decision trees, have demonstrated impressive capacities to improve the prediction accuracy of base learning algorithms. Further gains have been demonstrated by strategies that combine simple ensemble formation approaches. In this paper, we investigate the hypothesis that the improvement in accuracy of multi-strategy approaches to ensemble learning is due to an increase in the diversity of ensemble members that are formed. In addition, guided by this hypothesis, we develop three new multi-strategy ensemble learning techniques. Experimental results in a wide variety of natural domains suggest that these multi-strategy ensemble learning techniques are, on average, more accurate than their component ensemble learning techniques.

Taxonomy for characterizing ensemble methods in classification tasks: A review and annotated bibliography

“…It also offers the further advantage over AdaBoost of suiting parallel execution. MultiBoosting has been further extended by adding the stochastic attribute selection committee learning strategy to boosting and wagging [180]. The latter's research has shown that combining ensemble strategies would increase diversity at the cost of a small increase in individual test error resulting in a trade-off that reduced overall ensemble test error.…”

A Robust Ensemble Classification Method Analysis

“…Due to the different approaches applied to generate the committee, the decision trees in the final ensemble committee could be diverse from each other in certain ways. In the past decades, measuring diversity has become a very important issue in the research of Microarray ensemble classification methods [1,5,9].…”

“…There are also many statistical diversity measures available, such as diversity of errors [1,2,7], and pairwise and non-pairwise diversity measures [1,9,5]. It is desirable if every classifier in an ensemble committee can agree on most samples which are predicted correctly.…”

“…Up to date, all ensemble decision trees methods have kept diversity in mind [1,9]. However, among those methods, most of them, such as Boosting and Bagging, do not guarantee that each ensemble decision tree in the committee is different from outputs, namely identical trees and overlapping genes are not prohibited from an ensemble committee.…”

Ensemble learning methods for decision making: Status and future prospects

“…This flexibility in theory results in over fitting the training data than a single model. But, the practical ensemble techniques tend to reduce over fitting of the training data [3].…”