A survey of commonly used ensemble-based classification techniques

Abstract

The combination of multiple classifiers, commonly referred to as a classifier ensemble, has previously demonstrated the ability to improve classification accuracy in many application domains. As a result this area has attracted significant amount of research in recent years. The aim of this paper has therefore been to provide a state of the art review of the most well-known ensemble techniques with the main focus on bagging, boosting and stacking and to trace the recent attempts, which have been made to improve their performance. Within this paper, we present and compare an updated view on the different modifications of these techniques, which have specifically aimed to address some of the drawbacks of these methods namely the low diversity problem in bagging or the over-fitting problem in boosting. In addition, we provide a review of different ensemble selection methods based on both static and dynamic approaches. We present some new directions which have been adopted in the area of classifier ensembles from a range of recently published studies. In order to provide a deeper insight into the ensembles themselves a range of existing theoretical studies have been reviewed in the paper.

Impact of an Extra Layer on the Stacking Algorithm for Classification Problems

“…This may be explained by the limited number of level-1 heterogeneous classifiers (2) when compared to WEKA's standard number for homogeneous classifiers (10). Thus, this result corroborates with the literature (Jurek et al, 2014), in which a very small number of classifiers (heterogeneous classifiers grouped in pairs, determined by the research) in an ensemble has a worse average performance than those with several components (homogeneous classifiers that followed WEKA's standard parameters, in which 10 machines are created).…”

“…To solve this problem, Seewald (2002) Stacking efficiency is directly dependent on the number of classes of the problem (Jurek et al, 2014). A new approach called Troika was proposed by Menahem et al (2009) to address multi-class problems.…”

“…It is based on the four-layer architecture, where the last layer contained only one model: The super classifier, that outputs a vector of probabilities as a final decision of ensemble. Troika performed better than Stacking and StackingC in terms of classification accuracy (Jurek et al, 2014). This paper, differing from Troika's approach, will analyze the effects of an additional layer in the original Stacking algorithm, using only three levels (level-0, level-1 and level-2) and with only 2 meta-classifiers at level-1.…”

A dynamic model of classifier competence based on the local fuzzy confusion matrix and the random reference classifier

“…Combining outputs of these classifiers prevents the system from choosing the wrong classifier (Dietterich, 2000). The * Corresponding author idea of building ensemble systems has been being widely explored over the last two decades and it still has a great potential (Jurek et al, 2013;Dietterich, 2000;Wozniak et al, 2014). Multiclassifier systems proved to be an efficient tool for solving classification problems across domains such as bioinformatics (Plumpton, 2014;Fraz et al, 2012), economy (Hsieh and Hung, 2010) and many more (Wozniak et al, 2014).…”

Automatic Change Detection in High-Resolution Remote Sensing Images by Using a Multiple Classifier System and Spectral–Spatial Features

“…For the first stage, one possible strategy relies on using the same training data and different learning algorithms to build the base classifiers. Another approach is to build a set of base classifiers by using a single learning method and different training sets [30]. The main issue of this approach is the conversion of the original training set to obtain different training sets.…”