Evaluation of Lung MDCT Nodule Annotation Across Radiologists and Methods

Abstract

Rationale and Objectives-Integral to the mission of the National Institutes of Health-sponsored Lung Imaging Database Consortium is the accurate definition of the spatial location of pulmonary nodules. Because the majority of small lung nodules are not resected, a reference standard from histopathology is generally unavailable. Thus assessing the source of variability in defining the spatial location of lung nodules by expert radiologists using different software tools as an alternative form of truth is necessary.Materials and Methods-The relative differences in performance of six radiologists each applying three annotation methods to the task of defining the spatial extent of 23 different lung nodules were evaluated. The variability of radiologists' spatial definitions for a nodule was measured using both volumes and probability maps (p-map). Results were analyzed using a linear mixed-effects model that included nested random effects.Results-Across the combination of all nodules, volume and p-map model parameters were found to be significant at P < .05 for all methods, all radiologists, and all second-order interactions except one. The radiologist and methods variables accounted for 15% and 3.5% of the total p-map variance, respectively, and 40.4% and 31.1% of the total volume variance, respectively.Conclusion-Radiologists represent the major source of variance as compared with drawing tools independent of drawing metric used. Although the random noise component is larger for the p-map analysis than for volume estimation, the p-map analysis appears to have more power to detect differences in radiologist-method combinations. The standard deviation of the volume measurement task appears to be proportional to nodule volume.

The Lung Image Database Consortium (LIDC): An Evaluation of Radiologist Variability in the Identification of Lung Nodules on CT Scans

“…Such differences and the resulting variance in nodule volume and nodule margin characteristics provide the basis for on-going LIDC investigations; Meyer et al (21), for example, reported on the variance of nodule margins across six radiologists using three different software systems. When a radiologist assigned a lesion to the "nodule" category during the unblinded reading session, the radiologist provided a subjective assessment of nodule characteristics (nodule subtlety, internal structure, sphericity, margin, lobulation, spiculation, radiographic solidity, and likelihood of malignancy) through an electronic form.…”

The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans

“…The decision to allow multiple nodule outlining approaches was made after we conducted a study that demonstrated that the variation in nodule outlines derived from different radiologists substantially exceeded variation derived from different software tools. 33 One of the three systems, the one used by three institutions, uses a semiautomated technique 34 based on the Otsu method 35 to compute a threshold for region growing. The system also provides interactive editing tools including region addition, subtraction, and morphological operations.…”

The Lung Image Database Consortium (LIDC)

“…Meyer et al [23] took advantage of the boundaries drawn by six radiologists around 23 lung nodules to compute lung nodule volumes showing that radiologists' subjectivity is a major source of variability.…”

“…All of the 197 scans were acquired from multi-detector row CT scanners with pixel size ranging from 0.508 to 0.946 mm (average 0.66 mm) and an axial slice thickness ranging from 0.625 to 3.000 mm (average 1.7 mm, median 1.8 mm). The tube current ranged from 40 to 582 mA (average 177.5 mA, median 160 mA), tube voltage range was for more than half cases 120kVp with the remaining ones having voltages equal to 130kVp (8), 135kVp (23), and 140 kVp (28). All the processing was performed on anonymized data, devoid of any identifying information in accordance to the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule [14,15], that was provided by the LIDC institutions after approval by their respective IRBs.…”