Measuring Interobserver Disagreement in Rating Diagnostic Characteristics of Pulmonary Nodule Using the Lung Imaging Database Consortium and Image Database Resource Initiative

Lin, Hongli; Huang, Chang–Xing; Wang, Weisheng; Luo, Jiawei; Yang, Xuedong; Liu, Yuling

doi:10.1016/j.acra.2016.11.022

Cited by 9 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As nodules in adults are better delineated given the greater surrounding aeration, the measurement variability in children will likely be higher, which our data confirm (inter‐ and intra‐observer variability: 1.4–2.4 and 1.5–3.5 mm, respectively). Furthermore, identification of small lung nodules on chest CT is a common finding in otherwise healthy children 23 . Therefore, distinguishing benign from malignant lung nodules in children with malignant solid tumours is a real challenge considering the high prevalence of benign lesions and the considerable observer variability 24 .…”

Section: Discussionmentioning

confidence: 99%

“…Furthermore, identification of small lung nodules on chest CT is a common finding in otherwise healthy children. 23 Therefore, distinguishing benign from malignant lung nodules in children with malignant solid tumours is a real challenge considering the high prevalence of benign lesions and the considerable observer variability. 24 This was verified by a retrospective study that showed a substantial disagreement (benign vs. malignant) between biopsy results of lung nodules and the initial interpretation of the CT chest.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

The clinical impact of observer variability in lung nodule classification in children with Wilms tumour

Brok

Shelmerdine

Damsgaard

et al. 2022

Pediatric Blood & Cancer

View full text Add to dashboard Cite

Objectives: To investigate the extent to which observer variability of computed tomography (CT) lung nodule assessment may affect clinical treatment stratification in Wilms tumour (WT) patients, according to the recent Société Internationale d'Oncologie Pédiatrique Renal Tumour Study Group (SIOP-RTSG) UMBRELLA protocol. Methods: I: CT thoraces of children with WT submitted for central review were used to estimate size distribution of lung metastases. II: Scans were selected for blinded review by five radiologists to determine intra-and inter-observer variability. They assessed identical scans on two occasions 6 months apart. III: Monte Carlo simulation (MCMC) was used to predict the clinical impact of observer variation when applying the UMBRELLA protocol size criteria.Results: Lung nodules were found in 84 out of 360 (23%) children with WT. For 21 identified lung nodules, inter-observer limits of agreement (LOA) for the five readers were ±2.4 and ±1.4 mm (AP diameter), ±1.9 and ±1.8 mm (TS diameter) and ±2.0 and ±2.4 mm (LS diameter) at assessments 1 and 2. Intra-observer LOA across the three dimensions were ±1.5, ±2.2, ±3.5, ±3.1 and ±2.6 mm (readers 1-5). MCMC demonstrated that 17% of the patients with a 'true' nodule size of ≥3 mm will be scored as <3 mm, and 21% of the patients with a 'true' nodule size of <3 mm will be scored as being ≥3 mm. Conclusion:A significant intra-inter observer variation was found when measuring lung nodules on CT for patients with WT. This may have significant implications on treatment stratification, and thereby outcome, when applying a threshold of ≥3 mm for a lung nodule to dictate metastatic status.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

The clinical impact of observer variability in lung nodule classification in children with Wilms tumour

Brok

Shelmerdine

Damsgaard

et al. 2022

Pediatric Blood & Cancer

View full text Add to dashboard Cite

show abstract

“…This value corresponded to an integer ranging from 1 to 5 with the following designations related to the malignancy degree: (1) Highly Unlikely, (2) Moderately Unlikely, (3) Indeterminate, (4) Moderately Suspicious and (5) Highly Suspicious. The interobserver agreement in rating malignancy was previously studied and a modest agreement was found [25,26]. For cases rated by two radiologists, they only agree in 43% of the cases.…”

Section: Data Preparationmentioning

confidence: 98%

Pre-Training Autoencoder for Lung Nodule Malignancy Assessment Using CT Images

et al. 2020

View full text Add to dashboard Cite

Lung cancer late diagnosis has a large impact on the mortality rate numbers, leading to a very low five-year survival rate of 5%. This issue emphasises the importance of developing systems to support a diagnostic at earlier stages. Clinicians use Computed Tomography (CT) scans to assess the nodules and the likelihood of malignancy. Automatic solutions can help to make a faster and more accurate diagnosis, which is crucial for the early detection of lung cancer. Convolutional neural networks (CNN) based approaches have shown to provide a reliable feature extraction ability to detect the malignancy risk associated with pulmonary nodules. This type of approach requires a massive amount of data to model training, which usually represents a limitation in the biomedical field due to medical data privacy and security issues. Transfer learning (TL) methods have been widely explored in medical imaging applications, offering a solution to overcome problems related to the lack of training data publicly available. For the clinical annotations experts with a deep understanding of the complex physiological phenomena represented in the data are required, which represents a huge investment. In this direction, this work explored a TL method based on unsupervised learning achieved when training a Convolutional Autoencoder (CAE) using images in the same domain. For this, lung nodules from the Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) were extracted and used to train a CAE. Then, the encoder part was transferred, and the malignancy risk was assessed in a binary classification—benign and malignant lung nodules, achieving an Area Under the Curve (AUC) value of 0.936. To evaluate the reliability of this TL approach, the same architecture was trained from scratch and achieved an AUC value of 0.928. The results reported in this comparison suggested that the feature learning achieved when reconstructing the input with an encoder-decoder based architecture can be considered an useful knowledge that might allow overcoming labelling constraints.

show abstract

“…Lung nodule characterization is the most difficult step in the pipeline of lung cancer diagnosis according to radiologists, which can be observed by a great inter-observer disagreement on the task [7,12]. A lung nodule is normally characterized with respect to texture, spiculation, lobulation, and its morphological appearance on a CT scan, and eventually it must be classified as either benign or malignant for patient management.…”

Section: Introductionmentioning

confidence: 99%

The Pitfalls of Sample Selection: A Case Study on Lung Nodule Classification

Baltatzis¹,

Bintsi²,

Folgoc³

et al. 2021

Preprint

View full text Add to dashboard Cite

Using publicly available data to determine the performance of methodological contributions is important as it facilitates reproducibility and allows scrutiny of the published results. In lung nodule classification, for example, many works report results on the publicly available LIDC dataset. In theory, this should allow a direct comparison of the performance of proposed methods and assess the impact of individual contributions. When analyzing seven recent works, however, we find that each employs a different data selection process, leading to largely varying total number of samples and ratios between benign and malignant cases. As each subset will have different characteristics with varying difficulty for classification, a direct comparison between the proposed methods is thus not always possible, nor fair. We study the particular effect of truthing when aggregating labels from multiple experts. We show that specific choices can have severe impact on the data distribution where it may be possible to achieve superior performance on one sample distribution but not on another. While we show that we can further improve on the stateof-the-art on one sample selection, we also find that on a more challenging sample selection, on the same database, the more advanced models underperform with respect to very simple baseline methods, highlighting that the selected data distribution may play an even more important role than the model architecture. This raises concerns about the validity of claimed methodological contributions. We believe the community should be aware of these pitfalls and make recommendations on how these can be avoided in future work.

show abstract

Measuring Interobserver Disagreement in Rating Diagnostic Characteristics of Pulmonary Nodule Using the Lung Imaging Database Consortium and Image Database Resource Initiative

Cited by 9 publications

References 15 publications

The clinical impact of observer variability in lung nodule classification in children with Wilms tumour

The clinical impact of observer variability in lung nodule classification in children with Wilms tumour

Pre-Training Autoencoder for Lung Nodule Malignancy Assessment Using CT Images

The Pitfalls of Sample Selection: A Case Study on Lung Nodule Classification

Contact Info

Product

Resources

About