Objective: In multireader, multicase (MRMC) receiver operating characteristic (ROC) studies for evaluating medical imaging systems, the area under the ROC curve (AUC) is often used as a summary metric. Owing to the limitations of AUC, plotting the average ROC curve to accompany the rigorous statistical inference on AUC is recommended. The objective of this article is to investigate methods for generating the average ROC curve from ROC curves of individual readers. Methods: We present both a non-parametric method and a parametric method for averaging ROC curves that produce a ROC curve, the area under which is equal to the average AUC of individual readers (a property we call area preserving). We use hypothetical examples, simulated data and a real-world imaging data set to illustrate these methods and their properties. Results:We show that our proposed methods are area preserving. We also show that the method of averaging the ROC parameters, either the conventional bi-normal parameters (a, b) or the proper bi-normal parameters (c, d a ), is generally not area preserving and may produce a ROC curve that is intuitively not an average of multiple curves. Conclusion: Our proposed methods are useful for making plots of average ROC curves in MRMC studies as a companion to the rigorous statistical inference on the AUC end point. The software implementing these methods is freely available from the authors. Advances in knowledge: Methods for generating the average ROC curve in MRMC ROC studies are formally investigated. The area-preserving criterion we defined is useful to evaluate such methods.Multireader, multicase (MRMC) receiver operating characteristic (ROC) studies have been widely used to evaluate medical imaging devices and computer-assisted detection/ diagnosis devices in medical imaging. 1 Typically, multiple readers (i.e. radiologists) read images of multiple cases (i.e. patients), and, for each case, each reader rates his/her level of suspicion that a lesion is present based on the image. The diagnostic performance of each reader using an imaging device is thus characterized by a ROC curve constructed using his/her rating data. A ROC curve plots the true-positive fraction (TPF or sensitivity) as a function of the false-positive fraction (FPF or 1-specificity) as the decision threshold varies and thus illustrates the trade-off between the sensitivity and the specificity of a reader across all possible thresholds. 2The area under the ROC curve (AUC) is widely used to summarize the diagnostic performance of imaging systems.3 By analysing the sample AUCs that are obtained from a sample of readers reading a sample of cases, one makes statistical inference on the "population AUC", which represents the performance of the device expected (or averaged) over the population of readers and the population of cases. 4 As such, it characterizes the device performance itself independent of the particular readers and cases used in the study. Methods are well established for the design and analysis of such studies and tutorial...
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
customersupport@researchsolutions.com
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Copyright © 2025 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.