A Comparison of External Clustering Evaluation Indices in the Context of Imbalanced Data Sets

Abstract

National audienceFor highly imbalanced data sets, almost all the instances are labeled as one class, whereas far fewer examples are labeled as the other classes. In this paper, we present an empirical comparison of seven different clustering evaluation indices when used to assess partitions generated from highly imbalanced data sets. Some of the metrics are based on matching of sets (F-measure), information theory (normalized mutual information and adjusted mutual information), and pair of objects counting (Rand and adjusted Rand indices). We also investigate the BCubed metric, which takes into account the concepts of recall, precision, as well as counting pairs. Furthermore, in order to avoid the class size imbalance effect, we propose a modification to the Rand index, referred to as the normalized class size Rand (NCR) index. In terms of results, apart from NCR, our experiments indicate that all the other analyzed indices are not able to deal properly with the problem of class size imbalance

Adapted B-CUBED Metrics to Unbalanced Datasets

“…For each example, the first column shows the value obtained with the metric for the left partition, the second column shows the result for the right partition and the third column indicates if the formal constraint is satisfied ( ) or Note that none of current metrics can satisfy all constraints. Indeed, F b 3 satisfies the first 4 F.C., but misses the correct identification of the best partition for the unbalanced case as reported by [4]. However, the proposed modifications F mod&0.9 b 3 (with | x| = 3) and F 0.9 b 3 manage to correctly classify all the formal constraints using the parameter α = 0.9.…”

“…Finally, Cluster size vs. quantity gives higher scores to partitions where few clusters are provided but separates most classes. In addition to these formal constraints, the Unbalanced constraint was recently added by [4] and evaluates if a misclassification is present in a big class or in a small one. This constraint gives better scores when the incorrect classified element is from the biggest class.…”

“…This constraint gives better scores when the incorrect classified element is from the biggest class. Results using the examples proposed by [1] and [4] 4 are shown in Table 1 5 . For each example, the first column shows the value obtained with the metric for the left partition, the second column shows the result for the right partition and the third column indicates if the formal constraint is satisfied ( ) or Note that none of current metrics can satisfy all constraints.…”

“…Each of these constraints evaluate a different situation that must be solved with a good evaluation metric. However, in the particular case of unbalanced datasets, these metrics fail to identify the correct solution [4]. The particularity of an unbalanced dataset is that one of the classes covers most of the document collection.…”

Efficiency of Random Sampling Based Data Size Reduction on Computing Time and Validity of Clustering in Data Mining

“…Hence, for our research purpose the external indices could be more robust in comparison of clustering concordance between sample and complete datasets. In their research aiming to carry out the effect of sampling, de Souto et al (2012) also preferred to use the external validity indices for assessing the partitions for highly imbalanced datasets. In our study, since we expect that the cluster densities can be changed by the sampling rates we also assumed that the external indices would be more informative in comparison of the partitions obtained on different sample datasets.…”