Agreement of the Order of Overall Performance Levels Under Different Reading Paradigms

Abstract

Rationale and Objectives-To investigate consistency of the orders of performance levels when interpreting mammograms under three different reading paradigms.Materials and Methods-We performed a retrospective observer study in which nine experienced radiologists rated an enriched set of mammography examinations that they personally had read in the clinic ("individualized") mixed with a set that none of them had read in the clinic ("common set"). Examinations were interpreted under three different reading paradigms: binary using screening Breast Imaging Reporting and Data System (BI-RADS), receiver operating characteristic (ROC), and free-response ROC (FROC). The performance in discriminating between cancer and non-cancer findings under each of the paradigms were summarized using Youden's index/2+0.5 (Binary), nonparameteric area under the ROC curve (AUC), and an overall FROC index (JAFROC-2). Pearson correlation coefficients were then computed to assess consistency in the ordering of observers' performance levels. Statistical significance of the computed correlation coefficients was assessed using bootstrap confidence intervals obtained by re-sampling sets of examination-specific observations.Results-All but one of the computed pair-wise correlation coefficients were larger than 0.66 and were significantly different from zero. The correlation between the overall performance measures under the Binary and ROC paradigms was the lowest (0.43) and was not significantly different from zero (95% CI: from −0.078 to 0.733). Conclusion-The use of different evaluation paradigms in the laboratory tends to lead to consistent ordering of the overall performance levels of observers. However, one should recognize that conceptually similar performance indices resulting from different paradigms often measure different performance characteristics and thus disagreements are not only possible but frequently quite natural.

Correlation of free‐response and receiver‐operating‐characteristic area‐under‐the‐curve estimates: Results from independently conducted FROC/ROC studies in mammography

“…Interesting to note is that in a previous study by Gur et al, 53 where they investigated agreement of AUCs under FROC and ROC paradigms, an average correlation for all readers of about 0.7 (0.27-0.8; 95% confidence interval, CI) was found. This value is much higher than what we found in our study (0.32; 0.16-0.46; 95% CI).…”

Is an ROC-type Response Truly Always Better Than a Binary Response in Observer Performance Studies?

“…The difference between two AUCs of the entire ROC curve, and of a pair of operating points, depends on the location of these points along the curves and may be not only different in magnitude but also could be of opposite signs [21]. At the same time, there exist certain inequalities between the true values of these two indices.…”

When Radiologists Perform Best: The Learning Curve in Screening Mammogram Interpretation

“…We examined the effects of increasing years of breast imaging practice within individual radiologists, instead of comparing groups of radiologists with different years of experience. Another strength is that we examined performance in clinical practice, without having to rely on test sets, which might not accurately reflect clinical performance (29)(30)(31). Our detailed clinical information for each patient let us adjust radiologist performance measures for differences in case mix, and the statistical technique of conditional logistic regression controlled for any radiologist-level differences.…”