On the Dynamics of Classification Measures for Imbalanced and Streaming Data

Brzeziński, Dariusz; Stefanowski, Jerzy; Susmaga, Robert; Szczęch, Izabela

doi:10.1109/tnnls.2019.2899061

Cited by 46 publications

(37 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2 http://www.csie.ntu.edu.tw/∼cjlin/libsvmtools/datasets/ is a dataset with diverse weather patterns and concept drifts. Since the classification accuracy is not suitable for imbalanced data distribution, we adopt G-mean [67,68] as alternate metrics for evaluating the performance of classifiers in class imbalance scenarios.…”

Section: ) Real-world Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

Two-Stage Cost-Sensitive Learning for Data Streams With Concept Drift and Class Imbalance

Sun

Sun²,

Dai

2020

IEEE Access

View full text Add to dashboard Cite

Most methods for classifying data streams operate under the hypothesis that the distribution of classes is balanced. Unfortunately, the phenomenon of class imbalance widely exists in many real-world applications. In addition, the underlying concept of data stream may change in a certain way over time, and attacks increase the difficulty of data stream mining. Motivated by this challenge, a Two-Stage Cost-Sensitive (TSCS) classification is proposed for addressing the class imbalance issue in non-stationary data streams. We propose a novel two-stage cost-sensitive framework for data stream classification by utilizing cost information in both feature selection stage and classification stage. Moreover, a window adaptation and drift detection mechanism, which guarantees that an ensemble can adapt promptly to concept drift, is embedded in our method. Our algorithm is compared with competitive algorithms on different kinds of datasets. The result demonstrates that TSCS obtains significant improvement in terms of class imbalance data stream metrics.

show abstract

Section: ) Real-world Datasetsmentioning

confidence: 99%

“…G-mean is the geometric mean of the recall of abnormal classes and normal classes. Formally, the G-mean can be calculated as follows [67].…”

Section: ) Real-world Datasetsmentioning

confidence: 99%

Two-Stage Cost-Sensitive Learning for Data Streams With Concept Drift and Class Imbalance

Sun

Sun²,

Dai

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Concept drift has been thoroughly analyzed in the last two decades, in particular in the context of non-stationary data streams [18,26,33,40], resulting in drift taxonomies [79], detectors [26], evaluation techniques [84,85], and adaptive streaming classifiers [8]. Research on class imbalance has also led to many novel methods, such as class resampling [21], specialized classification methods [34] or dedicated classifier performance measures [11,12].…”

Section: Introductionmentioning

confidence: 99%

The impact of data difficulty factors on classification of imbalanced and concept drifting data streams

et al. 2021

Self Cite

View full text Add to dashboard Cite

Class imbalance introduces additional challenges when learning classifiers from concept drifting data streams. Most existing work focuses on designing new algorithms for dealing with the global imbalance ratio and does not consider other data complexities. Independent research on static imbalanced data has highlighted the influential role of local data difficulty factors such as minority class decomposition and presence of unsafe types of examples. Despite often being present in real-world data, the interactions between concept drifts and local data difficulty factors have not been investigated in concept drifting data streams yet. We thoroughly study the impact of such interactions on drifting imbalanced streams. For this purpose, we put forward a new categorization of concept drifts for class imbalanced problems. Through comprehensive experiments with synthetic and real data streams, we study the influence of concept drifts, global class imbalance, local data difficulty factors, and their combinations, on predictions of representative online classifiers. Experimental results reveal the high influence of new considered factors and their local drifts, as well as differences in existing classifiers’ reactions to such factors. Combinations of multiple factors are the most challenging for classifiers. Although existing classifiers are partially capable of coping with global class imbalance, new approaches are needed to address challenges posed by imbalanced data streams.

show abstract

“…A metric dependent on the prior (e.g. precision) will be affected by both differences indiscernibly [3] but a practitioner could be interested in isolating the variation of performance due to likelihood which reflects the intrinsic model's performance (see illustration in Fig. 1).…”

Section: Introductionmentioning

confidence: 99%

“…Precision-based metrics give false positives more importance, but they are tied to the class prior [2,3]. A new definition of precision and recall into precision gain and recall gain has been recently proposed to correct several drawbacks of AUC-PR [7].…”

Section: Introductionmentioning

confidence: 99%

Master Your Metrics with Calibration

Siblini¹,

Frery²,

He-Guelton³

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Machine learning models deployed in real-world applications are often evaluated with precision-based metrics such as F1-score or AUC-PR (Area Under the Curve of Precision Recall). Heavily dependent on the class prior, such metrics make it difficult to interpret the variation of a model's performance over different subpopulations/subperiods in a dataset. In this paper, we propose a way to calibrate the metrics so that they can be made invariant to the prior. We conduct a large number of experiments on balanced and imbalanced data to assess the behavior of calibrated metrics and show that they improve interpretability and provide a better control over what is really measured. We describe specific real-world use-cases where calibration is beneficial such as, for instance, model monitoring in production, reporting, or fairness evaluation.

show abstract

On the Dynamics of Classification Measures for Imbalanced and Streaming Data

Cited by 46 publications

References 29 publications

Two-Stage Cost-Sensitive Learning for Data Streams With Concept Drift and Class Imbalance

Two-Stage Cost-Sensitive Learning for Data Streams With Concept Drift and Class Imbalance

The impact of data difficulty factors on classification of imbalanced and concept drifting data streams

Master Your Metrics with Calibration

Contact Info

Product

Resources

About