On Ensemble Techniques for Data Stream Regression

Abstract

An ensemble of learners tends to exceed the predictive performance of individual learners. This approach has been explored for both batch and online learning. Ensembles methods applied to data stream classification were thoroughly investigated over the years, while their regression counterparts received less attention in comparison. In this work, we discuss and analyze several techniques for generating, aggregating, and updating ensembles of regressors for evolving data streams. We investigate the impact of different strategies for inducing diversity into the ensemble by randomizing the input data (resampling, random subspaces and random patches). On top of that, we devote particular attention to techniques that adapt the ensemble model in response to concept drifts, including adaptive window approaches, fixed periodical resets and randomly determined windows. Extensive empirical experiments show that simple techniques can obtain similar predictive performance to sophisticated algorithms that rely on reactive adaptation (i.e., concept drift detection and recovery).

Data stream analysis: Foundations, major tasks and tools

“…how predictions are combined and how diversity is induced. These were recently empirically analyzed in (Gomes, Montiel, Mastelini, Pfahringer, & Bifet, 2020).…”

Learning from evolving data streams through ensembles of random patches

“…The combination of random subspaces and instance sampling for data streams was recently explored in [22,23]. In [23], the authors presented theoretical insights related to ensembles of Hoeffding trees and several empirical experiments showing the advantage of combining online bagging and random subspaces with an active drift detection strategy.…”

“…In [23], the authors presented theoretical insights related to ensembles of Hoeffding trees and several empirical experiments showing the advantage of combining online bagging and random subspaces with an active drift detection strategy. In [22], the authors focused on applying random subspaces and online bagging for regression. The analysis was vastly empirical, and the conclusion was that local feature randomization [20] produced better results in terms of predictive performance when compared to global feature randomization schemes [23].…”

“…The analysis was vastly empirical, and the conclusion was that local feature randomization [20] produced better results in terms of predictive performance when compared to global feature randomization schemes [23]. Given the differences in nature of the task (regression vs classification), the results from [22] cannot be extended to the current work as we focus on classification. Also, we significantly extend the analysis presented in [23] by analytically explaining the impact of stable learners (such as Hoeffding trees) in a streaming setting.…”

“…In comparison with the paper that introduced Streaming Random Patches (SRP) [23], in this work we present an extended theoretical analysis; include an analysis of the impact of the random subspaces size and the resampling technique; and analyse how SRP and other ensembles perform when using Naive Bayes as the base learner. Recently, another work analysed the application of random patches for regression [22] in a streaming setting. This later work explored several empirical approaches to train base learners, combine their predictions and address concept drift.…”