What Your Radiologist Might be Missing: Using Machine Learning to Identify Mislabeled Instances of X-ray Images

Rädsch, Tim; Eckhardt, Sven; Leiser, Florian; Pandl, Konstantin D.; Thiebes, Scott; Sunyaev, Ali

doi:10.24251/hicss.2021.157

Cited by 11 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on the annotation decisions (for example, handling of occluded objects of interest) implemented in a dataset, ML practitioners need to define their own desired outcome for these occurrences and modify their ML model accordingly. Given their impact on the resulting annotations and the current poor state of datasets [6][7][8][9][10] , we argue that dataset creators and competition organizers should publish their labelling instructions, as proposed in ref. 39 .…”

Section: Discussionmentioning

confidence: 99%

“…Minimal text labelling instructions. 7). In this labelling instruction for example, the uncommon occurrence of text overlay is described in detail.…”

Section: Labelling Instructionsmentioning

confidence: 99%

“…High-quality annotated datasets are a critical bottleneck for supervised DL, and the quality of the annotated data is crucial for algorithm performance [2][3][4][5] . Recent work reflects an increasing awareness of widespread problems in commonly used image benchmarks, which are subject to errors 6,7 and biases 8,9 , and calls for a fundamental change in dataset culture 10 . Annotation-related problems may be particularly relevant in the field of biomedical image analysis, where data are typically sparse 11 , inter-rater variability is naturally high 12,13 , labelling ambiguities occur 14 and medical experts have their individual style of annotations 13,15 .…”

mentioning

confidence: 99%

See 2 more Smart Citations

Labelling instructions matter in biomedical image analysis

et al. 2023

Self Cite

View full text Add to dashboard Cite

Biomedical image analysis algorithm validation depends on high-quality annotation of reference datasets, for which labelling instructions are key. Despite their importance, their optimization remains largely unexplored. Here we present a systematic study of labelling instructions and their impact on annotation quality in the field. Through comprehensive examination of professional practice and international competitions registered at the Medical Image Computing and Computer Assisted Intervention Society, the largest international society in the biomedical imaging field, we uncovered a discrepancy between annotators’ needs for labelling instructions and their current quality and availability. On the basis of an analysis of 14,040 images annotated by 156 annotators from four professional annotation companies and 708 Amazon Mechanical Turk crowdworkers using instructions with different information density levels, we further found that including exemplary images substantially boosts annotation performance compared with text-only descriptions, while solely extending text descriptions does not. Finally, professional annotators constantly outperform Amazon Mechanical Turk crowdworkers. Our study raises awareness for the need of quality standards in biomedical image analysis labelling instructions.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Minimal text labelling instructions. 7). In this labelling instruction for example, the uncommon occurrence of text overlay is described in detail.…”

Section: Labelling Instructionsmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Labelling instructions matter in biomedical image analysis

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…Extensive studies on CXR imaging have shown their potential for severity assessment on the basis of lung involvement in the infection, disease progression detection [ 6 ], and prognosis prediction. However, in the case of COVID-19, the novelty of the disease makes it more challenging even for the expert radiologists to confidently interpret the findings, particularly on CXRs [ 12 , 13 , 14 ]. Therefore, an AI-based system that learns from expert radiologists and provides consistent results could be highly valuable in such situations.…”

Section: Related Workmentioning

confidence: 99%

Prognosis Prediction in COVID-19 Patients through Deep Feature Space Reasoning

et al. 2023

View full text Add to dashboard Cite

The COVID-19 pandemic has presented a unique challenge for physicians worldwide, as they grapple with limited data and uncertainty in diagnosing and predicting disease outcomes. In such dire circumstances, the need for innovative methods that can aid in making informed decisions with limited data is more critical than ever before. To allow prediction with limited COVID-19 data as a case study, we present a complete framework for progression and prognosis prediction in chest X-rays (CXR) through reasoning in a COVID-specific deep feature space. The proposed approach relies on a pre-trained deep learning model that has been fine-tuned specifically for COVID-19 CXRs to identify infection-sensitive features from chest radiographs. Using a neuronal attention-based mechanism, the proposed method determines dominant neural activations that lead to a feature subspace where neurons are more sensitive to COVID-related abnormalities. This process allows the input CXRs to be projected into a high-dimensional feature space where age and clinical attributes like comorbidities are associated with each CXR. The proposed method can accurately retrieve relevant cases from electronic health records (EHRs) using visual similarity, age group, and comorbidity similarities. These cases are then analyzed to gather evidence for reasoning, including diagnosis and treatment. By using a two-stage reasoning process based on the Dempster–Shafer theory of evidence, the proposed method can accurately predict the severity, progression, and prognosis of a COVID-19 patient when sufficient evidence is available. Experimental results on two large datasets show that the proposed method achieves 88% precision, 79% recall, and 83.7% F-score on the test sets.

show abstract

“…An important step is using datasets from multiple sources, or creating robust datasets from the start [Willemink et al, 2020] but this may not always be possible. However, existing datasets can still be critically evaluated for the presence of dataset shift [Rabanser et al, 2018], hidden subgroups (not reflected in the meta-data) , mislabeled instances [Rädsch et al, 2020] or other biases [Suresh and Guttag, 2019]. A checklist for structurally evaluating computer vision datasets for such problems is presented in [Zendel et al, 2017].…”

Section: Let Us Build Awareness Of Data Limitationsmentioning

confidence: 99%

How I failed machine learning in medical imaging -- shortcomings and recommendations

Varoquaux,

Cheplygina

2021

Preprint

View full text Add to dashboard Cite

Medical imaging is an important research field with many opportunities for improving patients' health. However, there are a number of challenges that are slowing down the progress of the field as a whole, such optimizing for publication. In this paper we reviewed several problems related to choosing datasets, methods, evaluation metrics, and publication strategies. With a review of literature and our own analysis, we show that at every step, potential biases can creep in. On a positive note, we also see that initiatives to counteract these problems are already being started. Finally we provide a broad range of recommendations on how to further these address problems in the future. For reproducibility, data and code for our analyses are available on https://github.com/GaelVaroquaux/ml med imaging failures.

show abstract

What Your Radiologist Might be Missing: Using Machine Learning to Identify Mislabeled Instances of X-ray Images

Cited by 11 publications

References 17 publications

Labelling instructions matter in biomedical image analysis

Labelling instructions matter in biomedical image analysis

Prognosis Prediction in COVID-19 Patients through Deep Feature Space Reasoning

How I failed machine learning in medical imaging -- shortcomings and recommendations

Contact Info

Product

Resources

About