AUC: A Better Measure than Accuracy in Comparing Learning Algorithms

Abstract

Abstract. Predictive accuracy has been widely used as the main criterion for comparing the predictive ability of classification systems (such as C4.5, neural networks, and Naive Bayes). Most of these classifiers also produce probability estimations of the classification, but they are completely ignored in the accuracy measure. This is often taken for granted because both training and testing sets only provide class labels. In this paper we establish rigourously that, even in this setting, the area under the ROC (Receiver Operating Characteristics) curve, or simply AUC, provides a better measure than accuracy. Our result is quite significant for three reasons. First, we establish, for the first time, rigourous criteria for comparing evaluation measures for learning algorithms. Second, it suggests that AUC should replace accuracy when measuring and comparing classification systems. Third, our result also prompts us to re-evaluate many well-established conclusions based on accuracy in machine learning. For example, it is well accepted in the machine learning community that, in terms of predictive accuracy, Naive Bayes and decision trees are very similar. Using AUC, however, we show experimentally that Naive Bayes is significantly better than the decision-tree learning algorithms.

Bug Prioritization to Facilitate Bug Report Triage

“…Eclipse and Gnome bug data was used for experiments. Results were evaluated using ROC (receiver operating characteristic) curve [24] . Performance of Multinomial Naïve Bayes was found to be better than that of other classification algorithms.…”

Evaluation of biometric systems: a study of users' acceptance and satisfaction

“…In this study, AUC is estimated using Mann-Whitney statistic test as presented by Ling et al (2003). The AUC of a classifier G is defined as:…”

“…Ling et al (2003) suggest that its use should replace accuracy when measuring and comparing classifiers: the best classifier is the one with the largest AUC value.…”

A study of users' acceptance and satisfaction of biometric systems

“…In our study, AUC is estimated using Mann-Whitney statistic test as presented in [38]. The AUC of a classifier G is defined as:…”

“…The authors in [38] suggest that its use should replace accuracy when measuring and comparing classifiers: the best classifier is the one with the largest AUC; • comprehensibility: qualifies the exploitability of the produced model. For example, in a Bayesian network, the important number of a node's parents affects the identification of its strong relations with them; • classification rapidity: which also would be a crucial factor if the training dataset is huge for example.…”