Pitfalls in Benchmarking Data Stream Classification and How to Avoid Them

Abstract

Abstract. Data stream classification plays an important role in modern data analysis, where data arrives in a stream and needs to be mined in real time. In the data stream setting the underlying distribution from which this data comes may be changing and evolving, and so classifiers that can update themselves during operation are becoming the state-of-the-art. In this paper we show that data streams may have an important temporal component, which currently is not considered in the evaluation and benchmarking of data stream classifiers. We demonstrate how a naive classifier considering the temporal component only outperforms a lot of current state-of-the-art classifiers on real data streams that have temporal dependence, i.e. data is autocorrelated. We propose to evaluate data stream classifiers taking into account temporal dependence, and introduce a new evaluation measure, which provides a more accurate gauge of data stream classifier performance. In response to the temporal dependence issue we propose a generic wrapper for data stream classifiers, which incorporates the temporal component into the attribute space.

Evolutionary model building under streaming data for classification tasks: opportunities and challenges

“…0 implies that the proposed model's predictions increasingly coincide with that of the naive model. Under classification tasks Bifet et al [21] note that assuming, say, a Naive Bayes classifier or even a simple majority class 'coin toss', would be informative for the case of data conforming to the i.i.d. assumption.…”

Concept Drift Detection with Hierarchical Hypothesis Testing

“…To further bridge the connections between our detection results and clustering results in [11], a recently developed measurement -Kappa Plus Statistic (KPS) [2,23] -have been proposed. KPS, defined as κ + = p0−p e 1−p e , aims to evaluate data stream classifier performance taken into account temporal dependence as well as the effectiveness (or rationality) of classifier adaptation, where p 0 is the classifier's prequential accuracy and p e is the accuracy of No-Change classifier.…”

Redundancies in Data and their Effect on the Evaluation of Recommendation Systems: A Case Study on the Amazon Reviews Datasets

“…Evaluation is from a scientific point of view not less but more important than the design of ever newer and "better" (really?) methods, and is typically challenging and far from trivial, as has been discussed for other areas of data mining as well [1,2,7,9,20,22].…”