Generating ensembles of heterogeneous classifiers using Stacked Generalization

Abstract

Over the last two decades, the machine learning and related communities have conducted numerous studies to improve the performance of a single classifier by combining several classifiers generated from one or more learning algorithms. Bagging and Boosting are the most representative examples of algorithms for generating homogeneous ensembles of classifiers. However, Stacking has become a commonly used technique for generating ensembles of heterogeneous classifiers since Wolpert presented his study entitled Stacked Generalization in 1992. Studies that have addressed the Stacking issue demonstrated that when selecting base learning algorithms for generating classifiers that are members of the ensemble, their learning parameters and the learning algorithm for generating the meta-classifier were critical issues. Most studies on this topic manually select the appropriate combination of base learning algorithms and their learning parameters. However, some other methods use automatic methods to determine good Stacking configurations instead of starting from these strong initial assumptions. In this paper, we describe Stacking and its variants and present several examples of application domains.

Stacked Regression With a Generalization of the Moore-Penrose Pseudoinverse

“…The methods should be also significantly diversified in order for the ensemble method to yield better results (Kuncheva and Whitaker (2003)). Noteworthy is also Table 2 in Sesmero et al (2015), where one can find information about base classifiers used in SR. The methods we selected are commonly used and meet the criteria of fast implementation and efficiency.…”

“…Although SR is applied to real-world problems less frequently than other ensemble methods, such as bagging or boosting, the exponential growth of data as well as the diversity of these data continue to make SR an interesting alternative (Sesmero et al (2015)). There are also some new papers which propose extensions to SR (Džeroski and Ženko (2004), Rooney et al (2004a), Rooney et al (2004b), Xu et al (2007), Ozay and Vural (2008), Ni et al (2009)), Ledezma et al (2010), Shunmugapriya and Kanmani (2013).…”

“…There are also some new papers which propose extensions to SR (Džeroski and Ženko (2004), Rooney et al (2004a), Rooney et al (2004b), Xu et al (2007), Ozay and Vural (2008), Ni et al (2009)), Ledezma et al (2010), Shunmugapriya and Kanmani (2013). An informative overview of SR methods can be found in Sesmero et al (2015). Sigletos et al (2005) pointed out that stacking using probabilities performs comparably or significantly better than voting.…”

A Detailed Analysis of Classifier Ensembles for Intrusion Detection in Wireless Network

“…Here, the base classifiers are called the level-0 classifiers, and the combiner is called meta-classifier [6,16]. One of the issue in stacking is obtaining the suitable base classifiers and the meta-classifier, especially in relation to each specific dataset [16].…”

“…1 shows the structure of stacking ensemble. To predict a new instance, the level-0 classifiers produce a vector of prediction that is the input to the meta-classifier, which in turn predict the class [16]. In this study, we use multi-response linear regression (MLR) as a meta-classifier [6,17].…”