Prequential AUC: properties of the area under the ROC curve for data streams with concept drift

Brzeziński, Dariusz; Stefanowski, Jerzy

doi:10.1007/s10115-017-1022-8

Cited by 108 publications

(68 citation statements)

References 38 publications

(93 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A quite recent study [30] introduces an efficient algorithm for calculating Prequential AUC , suitable for assessing classifiers on evolving data streams. Its statistical properties and comparison against simpler point measures, such as G-mean or Kappa statistics, has been examined in [33] .…”

Section: Evaluation Measuresmentioning

confidence: 99%

Ensemble learning for data stream analysis: A survey

et al. 2017

Self Cite

View full text Add to dashboard Cite

In many applications of information systems learning algorithms have to act in dynamic environments where data are collected in the form of transient data streams. Compared to static data mining, processing streams imposes new computational requirements for algorithms to incrementally process incoming examples while using limited memory and time. Furthermore, due to the non-stationary characteristics of streaming data, prediction models are often also required to adapt to concept drifts. Out of several new proposed stream algorithms, ensembles play an important role, in particular for non-stationary environments. This paper surveys research on ensembles for data stream classification as well as regression tasks. Besides presenting a comprehensive spectrum of ensemble approaches for data streams, we also discuss advanced learning concepts such as imbalanced data streams, novelty detection, active and semisupervised learning, complex data representations and structured outputs. The paper concludes with a discussion of open research problems and lines of future research.

show abstract

Section: Evaluation Measuresmentioning

confidence: 99%

Ensemble learning for data stream analysis: A survey

et al. 2017

Self Cite

View full text Add to dashboard Cite

show abstract

“…The microclusters with the highest number of points N is then moved to an empty set C to initialise a new cluster. After calculating its centre c, with Equation (20), and radius r, with Equation (21), the -neighbourhood method is again used to find density reachable microclusters. Among them, a process is undertaken to detect the so-called border microclusters [35] inside C, which obviously are not present during the first iteration as C initially contains only one microcluster.…”

Section: Detecting and Forming New Clustersmentioning

confidence: 99%

“…There are two fundamentals aspects to take into consideration in data stream clustering, namely concept drift and concept evolution. The first aspect refers to the phenomenon when the data in the stream undergo changes in the statistical properties of the clusters with respect to the time [19,20] while the second to the event when there is an unseen novel cluster appearing in the stream [5,21].…”

Section: Introductionmentioning

confidence: 99%

A Clustering System for Dynamic Data Streams Based on Metaheuristic Optimisation

et al. 2019

View full text Add to dashboard Cite

This article presents the Optimised Stream clustering algorithm (OpStream), a novel approach to cluster dynamic data streams. The proposed system displays desirable features, such as a low number of parameters and good scalability capabilities to both high-dimensional data and numbers of clusters in the dataset, and it is based on a hybrid structure using deterministic clustering methods and stochastic optimisation approaches to optimally centre the clusters. Similar to other state-of-the-art methods available in the literature, it uses "microclusters" and other established techniques, such as density based clustering. Unlike other methods, it makes use of metaheuristic optimisation to maximise performances during the initialisation phase, which precedes the classic online phase. Experimental results show that OpStream outperforms the state-of-the-art methods in several cases, and it is always competitive against other comparison algorithms regardless of the chosen optimisation method. Three variants of OpStream, each coming with a different optimisation algorithm, are presented in this study. A thorough sensitive analysis is performed by using the best variant to point out OpStream's robustness to noise and resiliency to parameter changes.

show abstract

“…However, AUC is only suitable for the offline learning situation. To improve the traditional AUC for online learning conditions, Brzezinski et al [29] modified the AUC and proposed the prequential area under the curve (PAUC). The value of the indicator is continually updated in the online learning situation.…”

Section: Evaluation Metricsmentioning

confidence: 99%

Online Learning Method for Drift and Imbalance Problem in Client Credit Assessment

Zhang

Liu

2019

Symmetry

View full text Add to dashboard Cite

Machine learning algorithms have been widely used in the field of client credit assessment. However, few of the algorithms have focused on and solved the problems of concept drift and class imbalance. Due to changes in the macroeconomic environment and markets, the relationship between client characteristics and credit assessment results may change over time, causing concept drift in client credit assessments. Moreover, client credit assessment data are naturally asymmetric and class imbalanced because of the screening of clients. Aiming at solving the joint research issue of concept drift and class imbalance in client credit assessments, in this paper, a novel sample-based online learning ensemble (SOLE) for client credit assessment is proposed. A novel multiple time scale ensemble classifier and a novel sample-based online class imbalance learning procedure are proposed to handle the potential concept drift and class imbalance in the client credit assessment data streams. The experiments are carried out on two real-world client credit assessment cases, which present a comprehensive comparison between the proposed SOLE and other state-of-the-art online learning algorithms. In addition, the base classifier preference and the computing resource consumption of all the comparative algorithms are tested. In general, SOLE achieves a better performance than other methods using fewer computing resources. In addition, the results of the credit scoring model and the Kolmogorov–Smirnov (KS) test also prove that SOLE has good practicality in actual client credit assessment applications.

show abstract

Prequential AUC: properties of the area under the ROC curve for data streams with concept drift

Cited by 108 publications

References 38 publications

Ensemble learning for data stream analysis: A survey

Ensemble learning for data stream analysis: A survey

A Clustering System for Dynamic Data Streams Based on Metaheuristic Optimisation

Online Learning Method for Drift and Imbalance Problem in Client Credit Assessment

Contact Info

Product

Resources

About