Measuring the Performance of Ordinal Classification

Abstract

Ordinal classi¯cation is a form of multiclass classi¯cation for which there is an inherent order between the classes, but not a meaningful numeric di®erence between them. The performance of such classi¯ers is usually assessed by measures appropriate for nominal classes or for regression. Unfortunately, these do not account for the true dimension of the error.The goal of this work is to show that existing measures for evaluating ordinal classi¯cation models su®er from a number of important shortcomings. For this reason, we propose an alternative measure de¯ned directly in the confusion matrix. An error coe±cient appropriate for ordinal data should capture how much the result diverges from the ideal prediction and how \ inconsistent" the classi¯er is in regard to the relative order of the classes. The proposed coe±cient results from the observation that the performance yielded by the Misclassi¯cation Error Rate coe±cient is the bene¯t of the path along the diagonal of the confusion matrix. We carry out an experimental study which con¯rms the usefulness of the novel metric.

Context-Sensitive Dynamic Ordinal Regression for Intensity Estimation of Facial Action Units

“…This score is obtained directly from a confusion matrix (CM). Given a normalized CM, OCI [46] is defined as…”

“…, where n r,c is the fraction (in %) of examples from the r-th class predicted as being from the c-th class, and the path is defined as a sequence of entries where two consecutive entries in the path are 8-adjacent neighbors (see [46] for details). For small values of β (we use 0.25), OCI focuses on measuring ordinal performance from CMs.…”

Context-Sensitive Conditional Ordinal Random Fields for Facial Action Intensity Estimation

“…The scores shown in the title of each graph are computed from the depicted sequences. Here we also include the Ordinal Classification Index (OCI) [3] score, whose lower values indicate less confusion among the neighboring levels. We see that the RVM model estimates well the slope of the true signal, but it misses its scale, which is a consequence of assuming an equal interval scale for the outputs.…”

Max-Ordinal Learning

“…, y K }, where y 1 ≺ · · · ≺ y K and ≺ is a linear order relation in Y. We use MAE as an example, but other options include using mean square error (MSE), average MAE (AMAE) or the ordinal classification index (OCI) [10].…”