Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)

Abstract

The receiver operating characteristic (ROC) has emerged as the gold standard for assessing and comparing the performance of classifiers in a wide range of disciplines including the life sciences. ROC curves are frequently summarized in a single scalar, the area under the curve (AUC). This article discusses the caveats and pitfalls of ROC analysis in clinical microarray research, particularly in relation to (i) the interpretation of AUC (especially a value close to 0.5); (ii) model comparisons based on AUC; (iii) the differences between ranking and classification; (iv) effects due to multiple hypotheses testing; (v) the importance of confidence intervals for AUC; and (vi) the choice of the appropriate performance metric. With a discussion of illustrative examples and concrete real-world studies, this article highlights critical misconceptions that can profoundly impact the conclusions about the observed performance.

Informational analysis: a Shannon theoretic approach to measure the performance of a diagnostic test

“…But this choice is acceptable, in clinical practice, only when SE or SP have high values; the informational approach and the figure 4 explain us why. (ii) from the classifier point of view, the best reasonable choice is to use the decision threshold which maximize the summation T P + T N [3], or something more elaborated, such as any objective function that is a linear combination of true and false positive rates via the convex hull [8]. (iii) another possible method is maximizing SE + SP , that is equivalent to the use of the so-called Youden index [9]; it uses the maximum vertical distance of ROC curve from the point (x, y) on the diagonal (chance) line.…”

“…Fallacy of the undistributed middle -ROC curves, even if widely used, have at least one significant pitfall, that is the so called "fallacy of the undistributed middle" [8]: all random models score an AUC of 0.5, but not every model that scores an AUC of 0.5 is random; in other words AU C = 0.5 does not necessarily imply that the classifier is no better than random guessing. An interesting example of this situation is given in figure 4a of [3], where the "twothresholds classifier" assumes that a certain quantity q, being below a threshold t 1 or exceeding a threshold t 2 > t 1 , indicates disease; while q ∈ [t 1 , t 2 ] means normal. Systolic blood pressure or expression of a gene could be an example of such a situation.…”

“…Systolic blood pressure or expression of a gene could be an example of such a situation. If we use ROC analysis, the AUC equals 0.5 (figure 4b of [3]), and so the AUC approach is not able to discriminate this good classifier from a random one. On the contrary, if one use the IRC approach, one can clearly discriminate this classifier, characterized by GIR = 0.342, from the random one (GIR = 0).…”

“…Different summary measures have been proposed to describe the accuracy with a single, objective index, e.g. the Area Under the Curve (AUC) provided by ROC analysis, and each of them shows well known advantages and drawbacks [3], [24]. Another important problem is which decision cut-off should be used to classify continuous test results, and how will the choice of a decision threshold affect comparisons between two diagnostic tests or between two raters [15].…”

An Empirical Evaluation of Ranking Measures With Respect to Robustness to Noise

“…It has been used for model selection in various applications, ranging from data mining competitions to biomedical tests (Berrar & Flach, 2012). The AUC is the area under the ROC curve, which depicts the tradeoffs between the false positive rate (or 1 minus specificity, depicted on the x-axis) and the true positive rate (or sensitivity, depicted on the y-axis).…”