Performance Analysis of Three-Class Classifiers: Properties of a 3-D ROC Surface and the Normalized Volume Under the Surface for the Ideal Observer

Abstract

Classification of a given observation to one of three classes is an important task in many decision processes or pattern recognition applications. A general analysis of the performance of three-class classifiers results in a complex six-dimensional (6D) receiver operating characteristic (ROC) space, for which no simple analytical tool exists at present. We investigate the performance of an ideal observer under a specific set of assumptions that reduces the 6D ROC space to 3D by constraining the utilities of some of the decisions in the classification task. These assumptions lead to a 3D ROC space in which the true-positive fraction (TPF) can be expressed in terms of the two types of false-positive fractions (FPFs). We demonstrate that the TPF is uniquely determined by, and therefore is a function of, the two FPFs. The domain of this function is shown to be related to the decision boundaries in the likelihood ratio plane. Based on these properties of the 3D ROC space, we can define a summary measure, referred to as the normalized volume under the surface (NVUS), that is analogous to the area under the ROC curve (AUC) for a two-class classifier. We further investigate the properties of the 3D ROC surface and the NVUS for the ideal observer under the condition that the three class distributions are multivariate normal with equal covariance matrices. The probability density functions (pdfs) of the decision variables are shown to follow a bivariate log-normal distribution. By considering these pdfs, we express the TPF in terms of the FPFs, and integrate the TPF over its domain numerically to obtain the NVUS. In addition, we performed a Monte Carlo simulation study, in which the 3D ROC surface was generated by empirical "optimal" classification of case samples in the multi-dimensional feature space following the assumed distributions, to obtain an independent estimate of NVUS. The NVUS value obtained by using the analytical pdfs was found to be in good agreement with that obtained from the Monte Carlo simulation study. We also found that, under all conditions studied, the NVUS increased when the difficulty of the classification task was reduced by changing the parameters of the class distributions, thereby exhibiting the properties of a performance metric in analogous to AUC. Our results indicate that, under the conditions that lead to our 3D ROC analysis, the performance of a 3-class classifier may be analyzed by considering the ROC surface, and its accuracy characterized by the NVUS.

The Validity of Three-Class Hotelling Trace (3-HT) in Describing Three-Class Task Performance: Comparison of Three-Class Volume Under ROC Surface (VUS) and 3-HT

“…Using medical imaging as an example, classic ROC analysis is limited in handling cases where more than two diagnostic alternatives are involved, etc. To address this limitation, many multiclass extensions of the classic ROC analysis have been proposed [1], [2], [4], [13], [18], [20], [33]. In addition to ROC-type approaches, many other approaches have been proposed to describe multiclass classification performance including the L-HT from multiclass LDA discussed in this work [24], [26].…”

“…In fact, the volume under the hypersurface, a natural extension of the binary FOM has been shown not to be useful [3]–[8], [14]. Sahiner reduced the 6-D space to a 3-D space by adding constraints on utility and introduced the normalized volume under surface (NVUS) as a FOM [18]. Everson introduced a multiclass ROC analysis method based on multiobjective optimization, and resulted in a generalization of the Gini coefficient as a FOM [9].…”

The Meaning and Use of the Volume Under a Three-Class ROC Surface (VUS)

“…However, multiclass ROC analysis is a theoretical problem whose solution has been eluded the community ever since the introduction of the binary ROC in the 1950s [19], [20]. Much work has been devoted to understanding the nature of a multiclass classification problem, and many metrics have been proposed to assess the performance of a multiclass classification task [12], [13], [19]- [32].…”

Anniversary Paper: History and status of CAD and quantitative image analysis: The role ofMedical Physicsand AAPM

“…115, [324][325][326][327] As decision making extends beyond twoclass diagnoses, n-class classifiers will require appropriate measures of performance, and these efforts are also being led by medical physicists. [328][329][330][331] Furthermore, during Jiang et al's research on CADx of clustered microcalcifications, the investigators realized the need for a more relevant measure of performance-beyond the area under the ROC curve ͑AUC͒-in situations such as diagnostic workup in which a high level of sensitivity is crucial, and, thus, the partial area index was developed as demonstrated in Fig. 8.…”