Building heterogeneous ensembles by pooling homogeneous ensembles

Abstract

Heterogeneous ensembles consist of predictors of different types, which are likely to have different biases. If these biases are complementary, the combination of their decisions is beneficial and could be superior to homogeneous ensembles. In this paper, a family of heterogeneous ensembles is built by pooling classifiers from M homogeneous ensembles of different types of size T. Depending on the fraction of base classifiers of each type, a particular heterogeneous combination in this family is represented by a point in a regular simplex in M dimensions. The M vertices of this simplex represent the different homogeneous ensembles. A displacement away from one of these vertices effects a smooth transformation of the corresponding homogeneous ensemble into a heterogeneous one. The optimal composition of such heterogeneous ensemble can be determined using cross-validation or, if bootstrap samples are used to build the individual classifiers, out-of-bag data. The proposed heterogeneous ensemble building strategy, composed of neural networks, SVMs, and random trees (i.e. from a standard random forest), is analyzed in a comprehensive empirical analysis and compared to a benchmark of other heterogeneous and homogeneous ensembles. The achieved results illustrate the gains that can be achieved by the proposed ensemble creation method with respect to both homogeneous ensembles and to the tested heterogeneous building strategy at a fraction of the training cost.

A Survey of Ensemble Learning: Concepts, Algorithms, Applications, and Prospects

“…Furthermore, parallel ensembles can be classified into homogeneous or heterogeneous, depending on the base learners' homogeneity. Homogeneous ensembles consist of models built using the same ML algorithm, while heterogeneous ensembles comprise models from different algorithms [46]- [48].…”

“…For example, heterogeneous ensembles employ different ML algorithms as base learners; therefore, they are essentially diverse. The main challenge in heterogeneous ensembles is obtaining the most effective method to combine the different base learners' predictions [46]. However, the main challenge of homogeneous ensemble methods is ensuring the base learners are diverse even though they use the same ML algorithm.…”

S-BEAM: A Semi-Supervised Ensemble Approach to Rank Potential Causal Variants and Their Target Genes in Microglia for Alzheimer’s Disease

“…Zhou (2009) described the benefits of the combination of both fields, particularly, they find that classifier combination can mitigate the instability caused by training with unlabeled data (Vries & Theirens, 2021). Notably, combined homogeneous-heterogeneous ensembles have demonstrated a better performance than the benchmark of solely one or the other (Sabzevari et al, 2022). In order to overcome the stochastic nature of individual classifiers, we incorporate homogeneous ensembles into our framework.…”

Heterogeneous Ensemble Deep Learning Model for Enhanced Arabic Sentiment Analysis

“…The combination of biased decisions could be superior to homogeneous ensembles. If these biases are complementary [ 16 ]. In most cases, Ensemble learning methods can be in the form of three popular ones, namely bagging [ 17 ], boosting [ 18 ], and stacking [ 19 ].…”