On the Binormal Predictive Receiver Operating Characteristic Curve for the Joint Assessment of Positive and Negative Predictive Values

Abstract

The predictive receiver operating characteristic (PROC) curve is a diagrammatic format with application in the statistical evaluation of probabilistic disease forecasts. The PROC curve differs from the more well-known receiver operating characteristic (ROC) curve in that it provides a basis for evaluation using metrics defined conditionally on the outcome of the forecast rather than metrics defined conditionally on the actual disease status. Starting from the binormal ROC curve formulation, an overview of some previously published binormal PROC curves is presented in order to place the PROC curve in the context of other methods used in statistical evaluation of probabilistic disease forecasts based on the analysis of predictive values; in particular, the index of separation (PSEP) and the leaf plot. An information theoretic perspective on evaluation is also outlined. Five straightforward recommendations are made with a view to aiding understanding and interpretation of the sometimes-complex patterns generated by PROC curve analysis. The PROC curve and related analyses augment the perspective provided by traditional ROC curve analysis. Here, the binormal ROC model provides the exemplar for investigation of the PROC curve, but potential application extends to analysis based on other distributional models as well as to empirical analysis.

Establishment and Characterization of an Empirical Biomarker SS/PV-ROC Plot Using Results of the UBC® Rapid Test in Bladder Cancer

“…However, evaluation of such diagnostics seems to be a complex procedure and clinicians will need assistance in learning how to deal with this approach with respect to the complexity of PV-ROC curves. Recently, articles published by Hughes [ 8 ] and Benish [ 9 ] provided valuable information and aid in dealing with this task (Benish does not discuss PV-ROC). A detailed evaluation of the findings in the presented article would include a discussion via information theory.…”

“…In this article, which is primarily focused on establishment of the empirical SS/PV-ROC plot, a detailed analysis via information theory is not included. Instead, the reader is referred to the articles of Hughes [ 8 ] and Benish [ 9 ] who reported on mutual information as a metric for predictive performance for PV-ROC and SS-ROC, respectively.…”

“…Unlike the SS-ROC, the form of PV-ROC is dependent on prevalence of cases in the dataset, and this has some impact on the extent to which any particular SS/PV-ROC can be generalized. PPV and 1−NPV increase as prevalence increases [ 8 ]. With respect to urinary bladder cancer, prevalence is known to differ in men and woman.…”

“…Furthermore, to obtain agreement on a standardized PV-ROC curve evaluation, it is important to make future empirical studies by different authors or institutions comparable. Recently published articles [ 8 , 9 ] will help to develop this new path relating to the diagnostics of bioassays for cancer and provide a base in other fields of science for general application of the SS/PV-ROC plot.…”

Mutual Information as a Performance Measure for Binary Predictors Characterized by Both ROC Curve and PROC Curve Analysis

“…In crop protection decision making, binary tests are disease predictors that provide a probabilistic risk assessment of, for example, epidemic vs. no epidemic, or treatment required vs. no treatment required. Context for the work described here is provided by four previous articles; in chronological order of publication, Vermont et al [ 1 ], Shiu and Gatsonis [ 2 ], Reibnegger and Schrabmair [ 3 ] and Hughes [ 4 ]. Vermont et al [ 1 ] described general strategies of threshold determination for both ROC curves and PROC curves.…”