The Reproducibility of Changes in Diagnostic Figures of Merit Across Laboratory and Clinical Imaging Reader Studies

Abstract

Rationale and Objectives In this paper we examine which comparisons of reading performance between diagnostic imaging systems made in controlled retrospective laboratory studies may be representative of what we observe in later clinical studies. The change in a meaningful diagnostic figure of merit between two diagnostic modalities should be qualitatively or quantitatively comparable across all kinds of studies. Materials and Methods In this meta-study we examine the reproducibility of relative sensitivity, false positive fraction, area under the ROC curve, and expected utility across laboratory and observational clinical studies for several different breast imaging modalities, including screen film mammography, digital mammography, breast tomosynthesis, and ultrasound. Results Across studies of all types the changes in the false positive fractions yielded very small probabilities of having a common mean value. The probabilities of relative sensitivity being the same across ultrasound and tomosynthesis studies were low. No evidence was found for different mean values of relative area under the ROC curve or relative expected utility within any of the study sets. Conclusion The comparison demonstrates that the ratios of areas under the ROC curve and expected utilities are reproducible across laboratory and clinical studies, whereas sensitivity and false positive fraction are not.

An Exploratory Multi-reader, Multi-case Study Comparing Transmission Ultrasound to Mammography on Recall Rates and Detection Rates for Breast Cancer Lesions

“…In an outside meta-analysis of 20 studies, ROC-AUC was shown to be an efficient way to simultaneously capture the performance of a device on cancer and non-cancer cases in MRMC studies across laboratories and a better method for predicting performance in clinical studies than cancer and noncancer recall rates (36). In line with this, it is the preferred accuracy statistical metric for comparisons of the U.S. Food and Drug Administration (37).…”

AAPM task group report 273: Recommendations on best practices for AI and machine learning for computer‐aided diagnosis in medical imaging

“…153,154 A common approach for assessing clinical performance is through a controlled reader study (either retrospective or prospective), directly comparing the performance of a human reader without and with output from the CAD-AI system. 155,156 A disadvantage of this approach is that the estimated performances are unlikely to match those in the true clinical setting because of differences in the cases, physicians, and reading process. It is important to realize that both the population of patients undergoing the examination (cases) and the population of physicians interpreting the data (readers) are sources of substantial variability in clinical reader studies.…”

“…A clinical reader performance assessment is used to estimate the clinical impact of a CAD‐AI algorithm 153,154 . A common approach for assessing clinical performance is through a controlled reader study (either retrospective or prospective), directly comparing the performance of a human reader without and with output from the CAD‐AI system 155,156 . A disadvantage of this approach is that the estimated performances are unlikely to match those in the true clinical setting because of differences in the cases, physicians, and reading process.…”