Influence of study design in receiver operating characteristics studies: sequential versus independent reading

Abstract

Abstract. Observer studies to assess new image processing devices or computer-aided diagnosis techniques are often performed, but little is known about the effect of the study design on observer performance results. We investigated the effect of the sequential and independent reading design on observer study results with respect to reader performance and their statistical power. For this we performed an observer study for the detection of lung nodules with bone-suppressed images (BSIs) compared with original chest radiographs. In a fully crossed observer study, eight observers assessed a series of 300 radiographs four times, including one assessment of the original radiograph with sequential BSI and two independent reading sessions with BSI. Observer performance was compared using multireader multicase receiver operating characteristics. No significant difference between the effect of BSI in the sequential and the independent reading sessions could be found (p ¼ 0.09; p ¼ 0.46). Compared with the original radiographs, increased performance with BSI was significant in the sequential and one of the independent reading sessions (p < 0.0001; p ¼ 0.0007), and nonsignificant in the other independent reading session (p ¼ 0.10). A strong increase of uncorrelated variance components was found in the independent reading sessions, masking the ability to demonstrate differences in observer performance across modalities. Therefore, the sequential reading design is the preferred design because it is less burdensome and has more statistical power.

Impact of Data Presentation on Physician Performance Utilizing Artificial Intelligence-Based Computer-Aided Diagnosis and Decision Support Systems

“…In addition, rather than simply comparing DS system performance to that of a radiologist, it extends these prior results by assessing workflows that more realistically approximate clinical practice. Past studies have shown mixed evidence on the effect of the tested reading methodologies on overall reader performance, but none have conducted an investigation within the context of diagnostic decision support, and their effects remain unknown [13]. Our results show a sizeable variation in performance obtained depending on which reading methodology was chosen (Table 4).…”

“…Prior research has explored the difference between sequential and independent study designs and their respective effects on performance [13–16]. These studies have suggested that both designs produce similar performance results within comparative analyses.…”

"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making

“…In the second phase (probe), we sought to understand users' information needs while making a decision with an AI Assistant. Participants diagnosed a prostate case with the aid of an AI Assistant: in line with sequential read procedure [64], they reviewed the case immediately before the algorithmic predictions were revealed. The algorithmic predictions (benign, grade 3, grade 4, and grade 5) were displayed as visual overlays on the image (Figure 1), which the pathologists could consider when making a final diagnostic decision.…”