Facing Imbalanced Data--Recommendations for the Use of Performance Metrics

Abstract

Recognizing facial action units (AUs) is important for situation analysis and automated video annotation. Previous work has emphasized face tracking and registration and the choice of features classifiers. Relatively neglected is the effect of imbalanced data for action unit detection. While the machine learning community has become aware of the problem of skewed data for training classifiers, little attention has been paid to how skew may bias performance metrics. To address this question, we conducted experiments using both simulated classifiers and three major databases that differ in size, type of FACS coding, and degree of skew. We evaluated influence of skew on both threshold metrics (Accuracy, F-score, Cohen's kappa, and Krippendorf's alpha) and rank metrics (area under the receiver operating characteristic (ROC) curve and precision-recall curve). With exception of area under the ROC curve, all were attenuated by skewed distributions, in many cases, dramatically so. While ROC was unaffected by skew, precision-recall curves suggest that ROC may mask poor performance. Our findings suggest that skew is a critical factor in evaluating performance metrics. To avoid or minimize skew-biased estimates of performance, we recommend reporting skew-normalized scores along with the obtained ones.

Feature enrichment through multi-gram models

“…In binary classifications, there are very few, if any, real-world scenarios where the data is evenly split between the positive and the negative category. The result of testing classifiers on highly skewed data that were created using balanced datasets, is generally the same: The performance drops for all threshold measures [15].…”

“…Instead, we will here look at the F-Measure values when comparing the threshold performance of the models [15]. In Table 5, to the left, we can see the tri-gram performance using imbalanced testing data.…”

“…Ensemble algorithms have been shown to improve performance on imbalanced data [16]. Finally, we will extend the tests from balanced testing to imbalanced testing with a data skew of roughly 20 as defined in [15], which means that in the final tests, the models with be tested using roughly 5% positive instances and 95% negative instances. That should give us an understanding of how the multi-gram models perform both on balanced and imbalanced data.…”

Exploring the Hidden Challenges Associated with the Evaluation of Multi-class Datasets Using Multiple Classifiers

“…Precision is the fraction of retrieved information relevant to the search while Recall is the fraction of the information related to the search query that is retrieved successfully [14,32]. The formulas for Sensitivity and Specificity are given below: …”

On the performance of classification algorithms for learning Pareto-dominance relations

“…ǫ = misclassified test instances total test instances , while the latter measures the speed of the algorithm in seconds. It has been argued in machine learning literature, whether or not misclassification rate is a good accuracy measure, especially when dealing with datasets having considerable class skew [21]. However, other performance measures [22] are equally susceptible to class skew [21].…”