Monte Carlo Validation of the Dorfman-Berbaum-Metz Method Using Normalized Pseudovalues and Less Data-Based Model Simplification1

Hillis, Stephen L.; Berbaum, Kevin S.

doi:10.1016/j.acra.2005.07.012

Cited by 64 publications

(44 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Selection of an image location, if a nodule was present in the image, was considered a true positive image identification. To compare the detection performance of the radiologists, a receiver operating characteristic (ROC) curve analysis based on the multiple reader multiple case (MRMC) method developed by Dorfman, Berbaum, and Metz and implemented in the software DBM MRMC 2.2 was used [16][17][18][19][20][21][22]. The analysis considered readers as fixed effects and the cases as random effects.…”

Section: Performance Evaluationmentioning

confidence: 99%

The Effects of Ambient Lighting in Chest Radiology Reading Rooms

et al. 2012

View full text Add to dashboard Cite

Under typical dark chest radiography reading room conditions, a radiologist's pupils contract and dilate as their visual focus intermittently shifts between the high luminance monitor and the darker background wall, resulting in increased visual fatigue and degradation of diagnostic performance. A controlled increase of ambient lighting may minimize these visual adjustments and potentially improve comfort and accuracy. This study was designed to determine the effect of a controlled increase of ambient lighting on chest radiologist nodule detection performance. Four chest radiologists read 100 radiographs (50 normal and 50 containing a subtle nodule) under low (E01 lx) and elevated (E050 lx) ambient lighting levels on a DICOM-calibrated, medicalgrade liquid crystal display. Radiologists were asked to identify nodule locations and rate their detection confidence. A receiver operating characteristic (ROC) analysis of radiologist results was performed and area under ROC curve (AUC) values calculated for each ambient lighting level. Additionally, radiologist selection times under both illuminance conditions were determined. Average AUC values did not significantly differ (p>0.05) between ambient lighting levels (estimated mean difference0−0.03; 95% CI, (−0.08, 0.03)). Average selection times decreased or remained constant with increased illuminance. The most considerable decreases occurred for false positive identification times (35.4±18.8 to 26.2±14.9 s) and true positive identification times (29.7± 18.3 to 24.5±15.5 s). No performance differences were statistically significant. Study findings suggest that a controlled increase of ambient lighting within darkly lit chest radiology reading rooms, to a level more suitable for performance of common radiological tasks, does not appear to have a statistically significant effect on nodule detection performance.

show abstract

Section: Performance Evaluationmentioning

confidence: 99%

The Effects of Ambient Lighting in Chest Radiology Reading Rooms

et al. 2012

View full text Add to dashboard Cite

show abstract

“…Whether the test fracture appeared in the second or third examination was random, but the same order was used for both experimental conditions in all experiments. The 47 test fractures were presented on digital radiographs of the foot [15], ankle [4], tibia/fibula [2], knee [3], shoulder [4], arm [3], elbow [1], wrist [3], and hand/fingers [8]. The 97 normal examinations appearing in the second or third positions of the series (47 with the test fractures, and 54 as pairs to make 27 normal patients) were presented on digital radiographs of the foot [7], ankle [13], tibia/fibula [3], knee [20], pelvis [19], chest [15], shoulder [3], arm [3], elbow [5], wrist [2], and hand/fingers [7].…”

Section: Methodsmentioning

confidence: 99%

“…Our primary method of analysis used the new DBM MRMC ROC analysis (11)(12)(13)(14)(15)(16) fitting the discrete rating data with the contaminated binormal model, treating area under the ROC curve and sensitivity at specificity = 0.9 as measures of detection accuracy, and treating patients as a fixed factor, and readers as a random factor. The discrete scale was used because the subjective probability scale failed to provide additional operating points.…”

Section: Roc Analysis and Statistical Analysismentioning

confidence: 99%

Satisfaction of Search in Multitrauma Patients: Severity of Detected Fractures

et al. 2007

Self Cite

View full text Add to dashboard Cite

Purpose-Satisfaction of search (SOS) occurs when an abnormality is missed because another abnormality has been detected. This research studied whether the severity of a detected fracture determines whether subsequent fractures are overlooked.Materials and Methods-Each of seventy simulated multi-trauma patients presented examinations of three anatomic areas. Readers evaluated each patient under two experimental conditions: when the images of the first anatomic area included a fracture (the SOS condition), and when it did not (the control condition). The SOS effect was measured on detection accuracy for subtle test fractures presented on examinations of the second and third anatomic areas. In an experiment with twelve radiology readers, the initial SOS radiographs showed non-displaced fractures of extremities, fractures associated with low morbidity. In another experiment with twelve different radiology readers, the initial examination, usually a CT, showed cervical and pelvic fractures of the type associated with high morbidity. Because of their more direct role in patient care, the experiment using high morbidity SOS fractures was repeated with seventeen orthopedic readers.Results-Detection of subtle test fractures was substantially reduced when fractures of low morbidity were added (p<0.01). No similar SOS effect was observed in either experiment in which added fractures were associated with high morbidity. Conclusion-The satisfaction of search effect in skeletal radiology was replicated, essentially doubling the evidence for SOS in musculoskeletal radiology, and providing an essential contrast to the absence of SOS from high morbidity fractures.

show abstract

“…Readers were made aware that contrast disks were randomly located in the image on approximately 66 % of the images. To identify a difference in detection performance based on different median background intensities, the reader data were analyzed using the Trapezoidal/Wilcoxon method in DBM-MRMC [20][21][22][23][24][25][26] 2.2 software (University of Chicago, Kurt Rossmann Laboratories for Radiologic Image Research, Chicago, IL, USA), by comparing the area under the ROC curve (AUC). Using JMP®, Version 9 statistical software (SAS Institute Inc., Cary, NC, USA), contingency tables were generated and used to determine the contrast threshold over all median backgrounds and to perform comparisons of reader scores at each background level.…”

Section: Methodsmentioning

confidence: 99%

Verification of DICOM GSDF in Complex Backgrounds

et al. 2012

View full text Add to dashboard Cite

While previous research has determined the contrast detection threshold in medical images, it has focused on uniform backgrounds, has not used calibrated monitors, or has involved a low number of readers. With complex clinical images, how the Grayscale Standard Display Function (GSDF) affects the detection threshold and whether the median background intensity shift has been minimized by GSDF remains unknown. We set out to determine if the median background affected the detection of a low-contrast object in a clustered lumpy background, which simulated a mammography image, and to define the contrast detection threshold for these complex images. Clustered lumpy background images were created of different median intensities and disks of varying contrasts were inserted. A reader study was performed with 17 readers of varying skill level who scored with a five-point confidence scale whether a disk was present. The results were analyzed using reader operating characteristic (ROC) methodology. Contingency tables were used to determine the contrast detection threshold. No statistically significant difference was seen in the area under the ROC curve across all of the backgrounds. Contrast detection fell below 50 % between +3 and +2 gray levels. Our work supports the conclusion that Digital Imaging and Communications in Medicine GSDF calibrated monitors do perceptually linearize detection performance across shifts in median background intensity. The contrast detection threshold was determined to be +3 gray levels above the background for an object of 1°visual angle.

show abstract

Monte Carlo Validation of the Dorfman-Berbaum-Metz Method Using Normalized Pseudovalues and Less Data-Based Model Simplification1

Cited by 64 publications

References 13 publications

The Effects of Ambient Lighting in Chest Radiology Reading Rooms

The Effects of Ambient Lighting in Chest Radiology Reading Rooms

Satisfaction of Search in Multitrauma Patients: Severity of Detected Fractures

Verification of DICOM GSDF in Complex Backgrounds

Contact Info

Product

Resources

About