From Captions to Explanations: A Multimodal Transformer-based Architecture for Natural Language Explanation Generation

Rio-Torto, Isabel; Cardoso, Jaime S.; Teixeira, Luís

doi:10.1007/978-3-031-04881-4_5

Cited by 1 publication

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In multi-label classification, the test radiological image is associated with multiple labels or concepts [20]- [30]. Most multi-label classification approaches, fine-tune a pre-trained CNN to predict if the images contain the given concepts present in the dataset [31]- [37]. Other approaches perform multi-label classification using deep features extracted from transformers [38]- [41].…”

Section: Related Workmentioning

confidence: 99%

Predicting Semantic Concepts in Heterogeneous Radiographic Images

Miranda,

Thenkanidiyoor,

Dinesh

2024

IEEE Access

View full text Add to dashboard Cite

Concept detection in a radiological image involves the identification of biomedical semantic entities within the given image. However, different modalities of radiological images make it difficult to design a single suitable approach that can handle this heterogeneity. Such imaging data also suffers from the problem of sparse concepts where several concepts are present in very few training images making it difficult to train machine learning models to correctly predict their occurrence. This paper proposes a hierarchical approach for concept detection in radiological images using deep features extracted from the layers of convolutional neural networks. At the first level, the modality of the radiological image is identified. The second level of classification detects concepts present in the input image using multi-label classification by considering only those concepts that are relevant to images belonging to the same modality as the input image. This multi-label classification is performed using suitable classifiers that efficiently handle underrepresented sparse concepts. The proposed hierarchical approach for concept detection in radiological images outperforms state-of-the-art methods for different datasets.INDEX TERMS Medical image modalities, radiological images, semantic concept detection, underrepresented sparse concepts.

show abstract

Section: Related Workmentioning

confidence: 99%