Binary and multi‐category ratings in a laboratory observer performance study: A comparison

Abstract

We investigated radiologists performances during retrospective interpretation of screening mammograms when using a binary decision whether to recall a woman for additional procedures or not and compared it with their receiver operating characteristic (ROC) type performance curves using a semi-continuous rating scale. Under an Institutional Review Board approved protocol nine experienced radiologists independently rated an enriched set of 155 examinations that they had not personally read in the clinic, mixed with other enriched sets of examinations that they had individually read in the clinic, using both a screening BI-RADS rating scale (recall/not recall) and a semi-continuous ROC type rating scale (0 to 100). The vertical distance, namely the difference in sensitivity levels at the same specificity levels, between the empirical ROC curve and the binary operating point were computed for each reader. The vertical distance averaged over all readers was used to assess the proximity of the performance levels under the binary and ROC-type rating scale. There does not appear to be any systematic tendency for the readers towards a better performance when using either of the two rating approaches, namely 4 readers performed better using the semi-continuous rating scale, 4 readers performed better with the binary scale, and one reader had the point exactly on the empirical ROC curve. Only one of the nine readers had a binary “operating point” that was statistically distant from the same reader’s empirical ROC curve. Reader-specific differences ranged from −0.046 to 0.128 with an average width of the corresponding 95% confidence intervals of 0.2 and p-values ranging for individual readers from 0.050 to 0.966. On average, radiologists performed similarly when using the two rating scales in that the average distance between the individual reader’s binary operating point and their ROC curve was close to zero. The 95% confidence interval for the fixed-reader average (0.016) was (−0.0206, 0.0631) (two-sided p-value 0.35). In conclusion we found that in retrospective observer performance studies the use of a binary response or a semi-continuous rating scale lead to consistent results in terms of performance as measured by sensitivity-specificity operating points.

Stereoscopic Interpretation of Low-Dose Breast Tomosynthesis Projection Images

“…A bootstrap analysis was carried out to ascertain this. This analysis was similar to what was described by Gur et al [9] in their work comparing mammography reader performances under binary and continuous/multicategory ratings. We summarize the key steps: Let us suppose that the ith reader is denoted by R i , and that for this reader and a given viewing mode (stereo or mono), the binary TPF and To assess whether the signed difference in sensitivities was significant or not, we performed bootstrap sampling.…”

“…The confidence scores, which indicated the reader's confidence in the presence of an abnormality, were collected for analyzing the readers' performance in an experimental setting using the receiver operating characteristic curve (ROC). Previous works [9] on analyzing binary and continuous/multi-category ratings in mammography observer studies have demonstrated that these two decision-making processes yield similar reader performances even though the binary true positive fraction (TPF) and false-positive fraction (FPF) operating points do not always lie on the ROC curves but in their vicinity.…”

Agreement of the Order of Overall Performance Levels Under Different Reading Paradigms

“…In a prior paper [30] we demonstrated an absence of systematic differences between the radiologist-specific Binary “operating point” and his/her ROC curve. That seemingly contradicting finding to the results of this paper is actually not a contradiction, rather a natural consequence of the differences in the performance measures used under the different paradigms.…”

“…In our previous paper [30] we experimentally demonstrated that reader-specific performances under the Binary paradigm do not exhibit systematic fluctuations from the ROC curves. That result experimentally supports the theoretical relationship often presumed to exist, at least on average, between the ROC curve and the operating point (sensitivity, specificity) estimated under the ROC and Binary paradigms.…”

Imaging Technology and Practice Assessments

“…Even though we found a considerable consistency among the performance levels of the readers regardless of the specific method or rating scale used (i.e., binary, receiver operating characteristic (ROC) or free-response ROC (FROC)) (2,3), our study showed that radiologists may perform significantly different in the clinic than in the laboratory when reading the very same cases. The differences were reflected in terms of both their overall performance levels (e.g.…”