Adaptive XGBoost for Evolving Data Streams

Abstract

Boosting is an ensemble method that combines base models in a sequential manner to achieve high predictive accuracy. A popular learning algorithm based on this ensemble method is eXtreme Gradient Boosting (XGB). We present an adaptation of XGB for classification of evolving data streams. In this setting, new data arrives over time and the relationship between the class and the features may change in the process, thus exhibiting concept drift. The proposed method creates new members of the ensemble from mini-batches of data as new data becomes available. The maximum ensemble size is fixed, but learning does not stop when this size is reached because the ensemble is updated on new data to ensure consistency with the current concept. We also explore the use of concept drift detection to trigger a mechanism to update the ensemble. We test our method on real and synthetic data with concept drift and compare it against batch-incremental and instance-incremental classification methods for data streams.

Application of Gradient Boosting Algorithms for Anti-money Laundering in Cryptocurrencies

“…ARF is an adaptation of RF, modified to work with evolving data-streams [18]. ARF proved to be an effective online model [7,35] and given that ASXGB is an extension of gradient boosting for online learning, an extension of RF was tested. The AXGB is an online adaptation of XGBoost [35], which inspired the development of ASXGB, and hence, it was fundamental to compare against this model.…”

“…ARF proved to be an effective online model [7,35] and given that ASXGB is an extension of gradient boosting for online learning, an extension of RF was tested. The AXGB is an online adaptation of XGBoost [35], which inspired the development of ASXGB, and hence, it was fundamental to compare against this model. In the original paper [35], two mechanisms were proposed to update the ensemble to handle concept drift: (a) push strategy: older models are removed before appending newer models, similar to First In First Out; (b) replace strategy: older models are replaced with newer ones [35].…”

“…We propose our own adaptation of the XGBoost algorithm [10] to handle evolving data-streams, coined as 'Adaptive Stacked eXtreme Gradient Boosting (ASXGB)'. A recent study published by Montiel et al [35] heavily influenced the work presented in this algorithm, as they showed that decision tree-based gradient boosting could effectively be adapted to create concept-drift-aware models. In this explanation, we describe our procedure in the context of binary classification, that is, y ∈ {C 1 , C 2 }.…”

“…To compare the proposed offline gradient boosting algorithms and our proposed online model, various benchmarks were tested alongside these algorithms. In the train/test split evaluation, an RF classifier was used as a benchmark Montiel et al [35] inspired the idea of buffering a datastream and splitting it into tumbling windows to train new XGBoost learners in the underlying ensemble. However, the update mechanism used to swap newer models within the ensemble and the prediction function differs from our solution, since we opted to combine the ensemble via stacking.…”

Flexible and Adaptive Fairness-aware Learning in Non-stationary Data Streams

“…The learning algorithms therefore need to take the evolution of underlying data distribution into consideration, while remaining stable on historical but not outdated concepts. Such adaptation is normally enabled by: 1) incorporating new instances from the stream into the model [32], [33], and ii) forgetting the previous outdated knowledge from the model [34], [35].…”