Observer studies involving detection and localization: Modeling, analysis, and validation

Abstract

Although the receiver operating characteristic (ROC) paradigm is the accepted method for evaluation of diagnostic imaging systems, it has some serious shortcomings inasmuch as it is restricted to one observer report per image. By contrast the free-response ROC (FROC) paradigm and associated analysis method allows the observer to report multiple abnormalities within each imaging study, and uses the location of reported abnormalities to improve the measurement. Because the ROC method cannot accommodate multiple responses or use location information, its statistical power will suffer. The FROC paradigm/analysis has not enjoyed widespread acceptance because of concern about whether responses made to the same diagnostic study can be treated as independent. We propose a new jackknife FROC analysis method (JAFROC) that does not make the independence assumption. The new analysis method combines elements of FROC and the Dorfman-Berbaum-Metz (DBM) methods. To compare JAFROC to an earlier free-response analysis method (specifically the alternative free-response, or AFROC method), and to the DBM method, which uses conventional ROC scoring, we developed a model for generating simulated FROC data. The simulation model is based on an eye-movement model of how experts evaluate images. It allowed us to examine null hypothesis (NH) behavior and statistical power of the different methods. We found that AFROC analysis did not pass the NH test, being unduly conservative. Both the JAFROC method and the DBM method passed the NH test, but JAFROC had more statistical power than the DBM method. The results of this comparison suggest that future studies of diagnostic performance may enjoy improved statistical power or reduced sample size requirements through the use of the JAFROC method.

Spatial Localization Accuracy of Radiologists in Free-Response Studies: Inferring Perceptual FROC Curves from Mark-Rating Data

“…These numbers allow one to plot a perceptual FROC operating point that is independent of the choice of acceptance-radius. In a multi-rating FROC study the most probable numbers of perceptual hits and misses for each cutoff could be analyzed by several methods (6,(9)(10)(11) to yield fitted FROC curves and estimates of observer performance. For multiple-reader multiple-case (MRMC) free-response studies the data could be analyzed along the lines of the Dorfman, Berbaum and Metz (DBM) method (37) to perform significance testing of the differences in performance between modalities.…”

“…In the ROC paradigm the data-unit is a single rating per image and no location information is collected. Several methods of analyzing free-response data have been described (6,(9)(10)(11)15,16). All of these methods share a well-known weakness that is detailed below.…”

“…For example, any mark made within the boundary of a lesion could be considered a LL (8). Another possibility is to select a distance criterion, termed acceptanceradius by us (6), and if a mark falls within this distance of the center of a lesion then it is classified as a LL and all other marks are classified as NLs. The choice of the acceptance-radius has an obvious effect on the observer's performance.…”

Worth a quick look? Initial scene previews can guide eye movements as a function of domain-specific expertise but can also have unforeseen costs.

“…In contrast, lung nodules are notoriously difficult to correctly identify, even in free viewing tasks. It is therefore standard practice in medical imaging tasks to obtain a confidence rating for each decision so that such information can be used in receiver operating characteristic (ROC) analysis to more accurately characterise performance (Chakraborty, & Berbaum, 2004;Green & Swets, 1966). Accordingly, once a participant had identified what they thought to be a nodule by pressing the gamepad button whilst looking at the suspected target, before proceeding to the next image participants were required to provide a 1-4 confidence rating (4 being highly confident) regarding their decision.…”

A ‘snapshot’ of the visual search behaviours of medical sonographers

“…The jack‐knifing free response operating characteristic method (JAFROC) extends upon ROC by taking lesion location into account. JAFROC has a range of 0–1 and it is defined as the probability that an obvious lesion seen on an abnormal image will be rated higher than the highest rated non‐lesion on a normal image 11 . JAFROC has been used widely for the analysis of human observer, free‐response data and validated in radiological populations 9 …”