A multimodal transformer to fuse images and metadata for skin disease classification

Cai, Gan; Zhu, Yu; Wu, Yue; Jiang, Xiaoben; Ye, Jiongyao; Yang, Dawei

doi:10.1007/s00371-022-02492-4

Cited by 58 publications

(29 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…According to some reviewed works, different feature extraction methods can influence the fusion results significantly. For example, Cai et al [51] observed that the ViT-based image encoder led to better fusion results than CNN-based encodes, and the fusion model using clinical features with soft one-hot encoding also outperformed hard encoding and word2vec. Li et al [58] compared the contribution of different pre-trained language models to multimodal fusion.…”

Section: Discussion and Future Workmentioning

confidence: 99%

“…Chen et al [26] calculated the co-attention weight to generate the genomic-guided WSI embeddings. Similarly, Lu et al [69] proposed a symmetric cross attention to fuse the genomic data and pathology image embeddings of glioma tumors for multitask learning, while Cai et al [51] proposed an asymmetrical multi-head cross attention to fuse the camera images and metadata for skin classification. Li et al [31] aggregated multiscale pathology images and clinical features to predict the lymph node metastasis (LNM) of breast cancer.…”

Section: Attention-based Fusion Methodsmentioning

confidence: 99%

“…Also, most of the camera images are three-channel color images but in a smaller size than pathology images. 2D pre-trained CNN networks [42] or pre-trained transformer models [51] are usually applied to the whole images or detected lesions.…”

Section: Camera Imagesmentioning

confidence: 99%

“…In preprocessing, categorical clinical features were usually converted through one-hot encoding, while the numerical features were standardized [16,34,35]. Specially, Cai et al [51] used soft one-hot encoding by setting the elements that were as 0 in the standard one-hot encoding as 0.1 to make contributions to the propagation. As the genomic data are in high dimension, some feature selection methods such as the highest variance were used to extract the most informative features [44,47,67].…”

Section: Structured Datamentioning

confidence: 99%

See 3 more Smart Citations

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

et al. 2023

View full text Add to dashboard Cite

The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on various image (e.g., radiological, pathological and camera image) and non-image data (e.g., clinical data andgenomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multi-modal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do we extract and aggregate multi-modal information to ultimately provide more objective, quantitative computer-aided clinical decision making? This paper reviews the recent studies on dealing with such a question. Briefly, this review will include the (1) overview of current multi-modal learning workflows, (2) summarization of multi-modal fusion methods, (3) discussion of the performance, (4) applications in disease diagnosis and prognosis, and (5) challenges and future directions.

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Section: Attention-based Fusion Methodsmentioning

confidence: 99%

Section: Camera Imagesmentioning

confidence: 99%

Section: Structured Datamentioning

confidence: 99%

See 2 more Smart Citations

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

et al. 2023

View full text Add to dashboard Cite

show abstract

“…To better utilize information from multiple sources, multi-modal fusion classification is introduced into the detection and classification of skin cancer. Cai et al ( 12 ) proposed a multimodal transformer to fuse multimodal information. Chen et al ( 13 ) proposed a skin cancer Multimodal Data Fusion Diagnosis Network (MDFNet) framework based on a data fusion strategy to effectively fuse clinical skin images with patient clinical data.…”

Section: Introductionmentioning

confidence: 99%

A deep learning based multimodal fusion model for skin lesion diagnosis using smartphone collected clinical images and metadata

Zhou

Yang

et al. 2022

Front. Surg.

View full text Add to dashboard Cite

IntroductionSkin cancer is one of the most common types of cancer. An accessible tool to the public can help screening for malign lesion. We aimed to develop a deep learning model to classify skin lesion using clinical images and meta information collected from smartphones.MethodsA deep neural network was developed with two encoders for extracting information from image data and metadata. A multimodal fusion module with intra-modality self-attention and inter-modality cross-attention was proposed to effectively combine image features and meta features. The model was trained on tested on a public dataset and compared with other state-of-the-art methods using five-fold cross-validation.ResultsIncluding metadata is shown to significantly improve a model's performance. Our model outperformed other metadata fusion methods in terms of accuracy, balanced accuracy and area under the receiver-operating characteristic curve, with an averaged value of 0.768±0.022, 0.775±0.022 and 0.947±0.007.ConclusionA deep learning model using smartphone collected images and metadata for skin lesion diagnosis was successfully developed. The proposed model showed promising performance and could be a potential tool for skin cancer screening.

show abstract

GIGO, Garbage In, Garbage Out: An Urban Garbage Classification Dataset

Sukel

Rudinac²,

Worring³

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

A multimodal transformer to fuse images and metadata for skin disease classification

Cited by 58 publications

References 32 publications

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

A deep learning based multimodal fusion model for skin lesion diagnosis using smartphone collected clinical images and metadata

GIGO, Garbage In, Garbage Out: An Urban Garbage Classification Dataset

Contact Info

Product

Resources

About