Boosting classifiers for drifting concepts

Abstract

This paper proposes a boosting-like method to train a classifier ensemble from data streams. It naturally adapts to concept drift and allows to quantify the drift in terms of its base learners. The algorithm is empirically shown to outperform learning algorithms that ignore concept drift. It performs no worse than advanced adaptive time window and example selection strategies that store all the data and are thus not suited for mining massive streams. 1 A preliminary short version of this paper was presented at the ECML/PKDD-2005 Workshop on Knowledge Discovery from Data Streams [47].

Adaptive cascade of boosted ensembles for face detection in concept drift

“…In these cases, both the structure of an ensemble and the weights of existing classifiers are altered. Wang et al [2] and Scholz and Klinkenberg [15] recognize the high update costs associated with continuous global updates of a model in highdensity data streams. They devise a framework for rapidly revising only relevant components of an ensemble and initiate the training of new classifiers only when required.…”

“…Ensemble-based techniques have been shown to be an effective and scalable approach to addressing conceptdrifting challenges in streaming data [11,[13][14][15]. Ensemble-based systems consist of multiple classifiers that can be viewed as a committee of experts, whose individual votes are combined in order to formulate a final classification.…”

Evolutionary model building under streaming data for classification tasks: opportunities and challenges

“…However, recent research has concentrated on approaches that actively grow/replace ensemble membership. Moreover, it may even be possible to maintain multiple ensembles in order to: (1) react to cyclic behaviours in the stream [158]; or (2) explicitly maintain multiple ensembles with different diversity properties [138]. A summary of potential design decisions associated with ensembles as applied to streaming data might therefore include:…”

“…• Incremental change to current knowledge (e.g., [107,158]) as opposed to the outright dropping of previous knowledge (e.g., [68,167]); • Adapting the weight associated with learners versus no weighting: Weight adaptation represents an intermediate level of refinement in which the models denoting the ensemble remain unchanged but their relative contribution to the voting is modified [2,90,152]. Conversely, weightless frameworks emphasize plasticity and tend to drop weaker ensemble members immediately (e.g., [81,167]); • Classifier weight adaption versus data instance based weight adaptation: The weighing of votes from an ensemble is generally a function of either classifier performance [54,65,152] or of the data from which a member of the ensemble was constructed (e.g., [20,150]); • Identification of ensemble member for replacement: Various heuristics have been proposed for targeting the ensemble member for replacement when performance as a whole is deemed to be poor, e.g., replace the oldest [167] or member with least 'contribution' [107,171].…”

“…The diversity term can be characterized as being 'good' or 'bad' [26], a result that has a corresponding observation in EC [187]. Constructing new ensemble member Diversity of base learner [65,102] Sample stream data using Boosting versus Bagging [147,150] Identify ensemble member for replacement Age based heuristics [167] Performance based heuristics [107,171] Class imbalance Effect of sampling biases [34,55,86,186] Drift management Incremental updating of current models [107,158] Adapt voting weights [2,90,152] Shift management Outright replacement of one or more ensemble member [68,81,167] Diversity management Impact on capacity for change [26,137,165] Genet Program Evolvable Mach Under non-stationary data, it has been established that reducing the absolute value for the ensemble margin produces an equivalent increase in diversity [165]. A second open question is in regard to the method assumed for combining the outcome from multiple models under a non-stationary task.…”