The problem of real-time extraction of meaningful patterns from timechanging data streams is of increasing importance for the machine learning and data mining communities. Regression in time-changing data streams is a relatively unexplored topic, despite the apparent applications. This paper proposes an efficient and incremental stream mining algorithm which is able to learn regression and model trees This paper has its origins in two conference papers that propose and partly evaluate a new algorithm for learning model trees from stationary data streams (Ikonomovska and Gama 2008) and an improvement of this algorithm for learning from non-stationary data streams (Ikonomovska et al. 2009). However, this paper significantly extends and upgrades the work presented there, both on the algorithmic design and experimental evaluation fronts. Concerning the algorithm: We consider a new approach for the split selection criteria; We include a memory saving method for disabling bad split points; We improve the adaptation mechanism by using the Q statistic with a fading factor (Gama et al. 2009); We discuss and propose several memory management methods that enable most effective learning with constrained resources (deactivating and reactivating non-problematic leaves, removing non-promising alternate trees). We also perform a much more extensive and in-depth experimental evaluation: We consider a larger collection of real-world datasets; we use a more carefully designed experimental methodology (sliding window prequential and holdout evaluation among others); we provide a much more comprehensive discussion of the experimental results. 123Learning model trees from evolving data streams 129 from possibly unbounded, high-speed and time-changing data streams. The algorithm is evaluated extensively in a variety of settings involving artificial and real data. To the best of our knowledge there is no other general purpose algorithm for incremental learning regression/model trees able to perform explicit change detection and informed adaptation. The algorithm performs online and in real-time, observes each example only once at the speed of arrival, and maintains at any-time a ready-to-use model tree. The tree leaves contain linear models induced online from the examples assigned to them, a process with low complexity. The algorithm has mechanisms for drift detection and model adaptation, which enable it to maintain accurate and updated regression models at any time. The drift detection mechanism exploits the structure of the tree in the process of local change detection. As a response to local drift, the algorithm is able to update the tree structure only locally. This approach improves the any-time performance and greatly reduces the costs of adaptation.

No abstract

a b s t r a c tThe emergence of ubiquitous sources of streaming data has given rise to the popularity of algorithms for online machine learning. In that context, Hoeffding trees represent the state-of-the-art algorithms for online classification. Their popularity stems in large part from their ability to process large quantities of data with a speed that goes beyond the processing power of any other streaming or batch learning algorithm. As a consequence, Hoeffding trees have often been used as base models of many ensemble learning algorithms for online classification. However, despite the existence of many algorithms for online classification, ensemble learning algorithms for online regression do not exist. In particular, the field of online any-time regression analysis seems to have experienced a serious lack of attention. In this paper, we address this issue through a study and an empirical evaluation of a set of online algorithms for regression, which includes the baseline Hoeffding-based regression trees, online option trees, and an online least mean squares filter. We also design, implement and evaluate two novel ensemble learning methods for online regression: online bagging with Hoeffding-based model trees, and an online RandomForest method in which we have used a randomized version of the online model tree learning algorithm as a basic building block. Within the study presented in this paper, we evaluate the proposed algorithms along several dimensions: predictive accuracy and quality of models, time and memory requirements, bias-variance and bias-variance-covariance decomposition of the error, and responsiveness to concept drift.

No abstract