Visual-based analysis of classification measures and their properties for class imbalanced problems

Abstract

With a plethora of available classification performance measures, choosing the right metric for the right task requires careful thought. To make this decision in an informed manner, one should study and compare general properties of candidate measures. However, analysing measures with respect to complete ranges of their domain values is a difficult and challenging task. In this study, we attempt to support such analyses with a specialized visualization technique, which operates in a barycentric coordinate system using a 3D tetrahedron. Additionally, we adapt this technique to the context of imbalanced data and put forward a set of properties which should be taken into account when selecting a classification performance measure. As a result, we compare 22 popular measures and show important differences in their behaviour. Moreover, for parametric measures such as the F β and IBA α (G-mean), we analytically derive parameter thresholds that change measure properties. Finally, we provide an online visualization tool that can aid the analysis of complete domain ranges of performance measures. NOTICE: This is a preliminary version of an article submitted to Information Sciences all situations. However, which measure is used in a given problem seems to be, to a large extent, dictated simply by the measure's popularity rather than a thorough discussion of its properties.Although there are a few systematic studies on different properties of classifier performance measures [19,16,11,30], we still postulate the need for thorough analysis of the measures' behaviour. In particular, methods for: interpreting and comparing measures with respect to whole domain ranges, analysing their nature for different class and prediction distributions, and detecting the presence of unusual values are much needed. Theoretical investigations of these aspects are often very laborious and time consuming, especially when multi-dimensional aspects, provided by the confusion matrices, need to be taken into account. Due to these difficulties, such an analysis could be alternatively carried out with visual techniques to aid the understanding and interpretability of various measure properties.In this paper, we put forward a new visualization technique for analysing entire domains of classification performance measures, which depicts all possible configurations of predictions in a confusion matrix, regardless of the used classifier. For this purpose, we adapt an approach originally created for rule interestingness measures to the context of classification [31]. Contrary to existing performance measure visualizations, such as ROC space [11], the proposed approach presents measures in a space which is defined directly on elements of the confusion matrix, is easily interpretable in 3D, and remains defined for all elements of the domain. Moreover, based on the devised visualization, we propose ten properties which should be taken into account while selecting evaluation measures, particularly for class imbalanced data. Consequently, we compare 22 popular c...

Analysing the Effect of Platform and Operating System Features on Predicting Consumers’ Purchase Intent using Machine Learning Algorithms

“…Three well-known machine learning prediction models are used to explore the effect of users' platforms on their purchase intention behaviours. These (Brzezinski et al, 2018), (Bader-El-Den et al, 2018). Furthermore, in order to validate the proposed approach and evaluate the ability of the machine learning prediction models, we run experiments with ten cross-validations.…”

The impact of data difficulty factors on classification of imbalanced and concept drifting data streams

“…G-mean is the geometric mean of recall (TPR) and specificity (True Negative Rate-TNR), defined as G mean = √ TPR × TNR. Recall and G-mean were selected from a larger list of measures [11,34] mainly due to their complementary nature and easy interpretation. Recall focuses only on the minority class, allowing us to see when the recognition of the minority class drops.…”

“…For example, if Recall improves but G-mean deteriorates, this means that the recognition of the minority class has improved at the cost of the recognition rate of the majority class. Moreover, G-mean is skew-invariant, meaning that G-mean's interpretation remains the same for all possible class imbalance ratios [11,12], being particularly relevant for studying drifting imbalance ratios.…”

“…Concept drift has been thoroughly analyzed in the last two decades, in particular in the context of non-stationary data streams [18,26,33,40], resulting in drift taxonomies [79], detectors [26], evaluation techniques [84,85], and adaptive streaming classifiers [8]. Research on class imbalance has also led to many novel methods, such as class resampling [21], specialized classification methods [34] or dedicated classifier performance measures [11,12].…”

High-Resolution Remote Sensing Imagery Classification of Imbalanced Data Using Multistage Sampling Method and Deep Neural Networks

“…Because of the accuracy as an evaluation, metrics will make sense only if the class labels of test data are uniformly distributed. So in this paper the performance of the various methods was evaluated based on the criteria as following: per-class precision, overall accuracy (OA), average recall, average F 1 -score and G-mean, which are considered to be easily interpretable and have better theoretical properties than other classification measures for class imbalanced problems [56]. Mutual usage of the G-mean measure and overall accuracy is considered in the evaluation of sampling schema in order to achieve the optimal performance for both majority and minority classes.…”