Use of ROC curve analysis for prediction gives fallacious results: Use predictivity-based indices

Abstract

The area under the ROC curve is frequently used for assessing the predictive efficacy of a model, and the Youden index is commonly used to provide the optimal cut-off. Both are misleading tools for predictions. A ROC curve is drawn for the sensitivity of a quantitative test against its (1 – specificity) at different values of the test. Both sensitivity and specificity are retrospective in nature as these are indicators of correct classification of already known conditions. They are not indicators of future events and are not valid for predictions. Predictivity intimately depends on the prevalence which may be ignored by sensitivity and specificity. We explain this fallacy in detail and illustrate with several examples that the actual predictivity could differ greatly from the ROC curve-based predictivity reported by many authors. The predictive efficacy of a test or a model is best assessed by the percentage correctly predicted in a prospective framework. We propose predictivity-based ROC curves as tools for providing predictivities at varying prevalence in different populations. For optimal cut-off for prediction, in place of the Youden index, we propose a P-index where the sum of positive and negative predictivities is maximum after subtracting 1. To conclude, for correctly assessing adequacy of a prediction models, predictivity-based ROC curves should be used instead of the usual sensitivity-specificity-based ROC curves and the P-index should replace the Youden index.