Having a Blast: Meta-Learning and Heterogeneous Ensembles for Data Streams

Abstract

Ensembles of classifiers are among the best performing classifiers available in many data mining applications. However, most ensembles developed specifically for the dynamic data stream setting rely on only one type of base-level classifier, most often Hoeffding Trees. In this paper, we study the use of heterogeneous ensembles, comprised of fundamentally different model types. Heterogeneous ensembles have proven successful in the classical batch data setting, however they do not easily transfer to the data stream setting. We therefore introduce the Online Performance Estimation framework, which can be used in data stream ensembles to weight the votes of (heterogeneous) ensemble members differently across the stream. Experiments over a wide range of data streams show performance that is competitive with state of the art ensemble techniques, including Online Bagging and Leveraging Bagging. All experimental results from this work are easily reproducible and publicly available on OpenML for further analysis.

Classification of the drifting data streams using heterogeneous diversified dynamic class-weighted ensemble

“…For the data stream classification, a Heterogeneous Ensemble with Feature drift for Data Streams (HEFT-Stream) Nguyen et al (2012) builds a heterogeneous ensemble composed of different online classifiers (e.g., Online Naive Bayes). Adaptive modifications of the heterogeneous ensembles were also successfully applied on the drifting data streams ( Van Rijn et al, 2016 ; Frías-Blanco et al, 2016 ; Van Rijn et al, 2018 ; Idrees et al, 2020 ), and many of them proved suitable to address issues such as class imbalance ( Large, Lines & Bagnall, 2017 ; Fernández et al, 2018 ; Ren et al, 2018 ; Wang, Minku & Yao, 2018 ; Ghaderi Zefrehi & Altınçay, 2020 ). The approach described in this paper aims to combine the construction of the adaptive heterogeneous ensemble with a diversity-based update of the ensemble members.…”

A heterogeneous online learning ensemble for non-stationary environments

“…BLAST (short for best last) [42] introduced an Online Performance Estimation framework to weight the votes of (heterogeneous) ensemble members. Based on zero/one loss function, i.e.…”

Variational inference based bayes online classifiers with concept drift adaptation

“…Online Gradient Descent) and NOGD (Nystrom Online Gradient Descent) [2]; ensemble-based BLAST [3]; state-of-the-art second-order linear AROW (Adaptive Regularization of Weights) [4]; widely used first-order linear PA (Passive Aggressive learning) [5], the most used decision tree HT (Hoeffding Tree) [6]. We also compare our proposed methods with ONBG (Online Naïve Bayes for Gaussians) [7] a first-order generative method.…”

“…Bayesian framework (BE) [27] was proposed for recursively estimating the classifier weights in online learning of a classifier ensemble. Another recent interesting work, named as BLAST [3], studies the use of heterogeneous ensembles comprised of fundamentally different model types.…”

“…Five diverse online learning classifiers, namely most widely used first-order linear PA [5], stateof-the-art second-order linear AROW [4], the most-used decision tree HT [6], recent heterogeneous ensemble BLAST [3], and the simplest first-order generative ONBG (Online Naïve Bayes for Gaussians) (described in [7], however in our experiment we do not apply random projection). The code of HT, BLAST, PA and AROW is available in the library MOA…”