Multi-class ROC analysis from a multi-objective optimisation perspective

Abstract

The Receiver Operating Characteristic (ROC) has become a standard tool for the analysis and comparison of classifiers when the costs of misclassification are unknown. There has been relatively little work, however, examining ROC for more than two classes. Here we discuss and present an extension to the standard twoclass ROC for multi-class problems.We define the ROC surface for the Q-class problem in terms of a multi-objective optimisation problem in which the goal is to simultaneously minimise the Q(Q − 1) misclassification rates, when the misclassification costs and parameters governing the classifier's behaviour are unknown. We present an evolutionary algorithm to locate the Pareto front-the optimal trade-off surface between misclassifications of different types. The use of the Pareto optimal surface to compare classifiers is discussed and we present a straightforward multi-class analogue of the Gini coefficient. The performance of the evolutionary algorithm is illustrated on a synthetic three class problem, for both k-nearest neighbour and multi-layer perceptron classifiers.

Visualising High-Dimensional Pareto Relationships in Two-Dimensional Scatterplots

“…Neuroscale [20,19] has also been used for multi-objective visualisation [11,8] -but unlike PCA it provides a non-linear mapping. However, although popular across many application domains, both Neuroscale and PCA are oblivious to whether solutions dominate each other, or are mutually non-dominating in multi-objective populations, or what their Pareto shell is.…”

Optimizing Decision Trees Using Multi-objective Particle Swarm Optimization

“…Often this is not the case (for example when screening for cancers, or in safety related classification problems). Where the costs are unknown a priori and/or the shape of the trade-off front is unknown, it is appropriate to trade-off the different misclassification rates in parallel [15,16]. An illustration of this is provided in Figure 5 shows the objective space mapping of three decision trees, where the objectives are minimising the class 1 misclassification rate and minimising the class 2 misclassification rate.…”

“…Note this is equivalent to the widely used Receiver Operating Characteristic (ROC) curve representation used for binary classification tasks -by representing the curve in terms of both misclassification rates, instead of focusing on a single class (correct assignment and incorrect assignment rates to that class), the problem is more easily extended to multiple (i.e. > 2) class problems [15,16]. The class mapping on feature space caused by the three mutually non-dominating trees are also plotted in 5, and arranged in the order they are plotted in the trade-off front.…”

“…The Iris problem having 6 objectives is less easy to visualise in this format, so an alternative representation is provided in Figure 10, providing a histogram of the the distances of points on the trade-off front from the random allocation simplex [15,16]. The random allocation simplex is the plane in misclassification rate space which denotes classifiers which assign classes to data points at random (with some probability).…”

Model Assessment with ROC Curves

“…Robust tools such as the ROCR package 3 for the R environment (Sing, Sander, Beerenwinkel, & Lengauer, 2005) contribute to the rapid adoption of ROC analysis as the preferred model analysis technique. At a technical level, the most important development is the extension of this analysis technique from binary classification problems to multi-class problems providing a much wider applicability of this technique (Everson & Fieldsend, 2006;Lane, 2000;Srinivasan, 1999).…”