Interpretability-Based Multimodal Convolutional Neural Networks for Skin Lesion Diagnosis

Wang, Sutong; Yin, Yunqiang; Wang, Dujuan; Wang, Yanzhang; Jin, Yaochu

doi:10.1109/tcyb.2021.3069920

Cited by 56 publications

(24 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [ 35 ], a multimodal convolutional neural network (IM-CNN) is presented, a model for the multiclass classification of dermatoscopic images and patient metadata as input for diagnosing pigmented skin lesions. The modeling was carried out on the open dataset HAM10,000 (“Human versus machine with 10,000 training images”), part of the ISIC Melanoma Project open database, and consists of seven diagnostic categories.…”

Section: Discussionmentioning

confidence: 99%

“…In database dermatology, heterogeneous data mining makes it possible to combine patient statistical metadata and dermoscopic images, greatly improving the recognition of pigmented skin lesions. The use of multimodal neural network systems [ 34 , 35 , 36 , 37 ], as well as methods for combining metadata and multidimensional visual data [ 38 ], has significantly improved the accuracy in recognizing pigmented skin lesions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

System for the Recognizing of Pigmented Skin Lesions with Fusion and Analysis of Heterogeneous Data Based on a Multimodal Neural Network

Lyakhov

Lyakhova

Nagornov

2022

Cancers

View full text Add to dashboard Cite

Today, skin cancer is one of the most common malignant neoplasms in the human body. Diagnosis of pigmented lesions is challenging even for experienced dermatologists due to the wide range of morphological manifestations. Artificial intelligence technologies are capable of equaling and even surpassing the capabilities of a dermatologist in terms of efficiency. The main problem of implementing intellectual analysis systems is low accuracy. One of the possible ways to increase this indicator is using stages of preliminary processing of visual data and the use of heterogeneous data. The article proposes a multimodal neural network system for identifying pigmented skin lesions with a preliminary identification, and removing hair from dermatoscopic images. The novelty of the proposed system lies in the joint use of the stage of preliminary cleaning of hair structures and a multimodal neural network system for the analysis of heterogeneous data. The accuracy of pigmented skin lesions recognition in 10 diagnostically significant categories in the proposed system was 83.6%. The use of the proposed system by dermatologists as an auxiliary diagnostic method will minimize the impact of the human factor, assist in making medical decisions, and expand the possibilities of early detection of skin cancer.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

System for the Recognizing of Pigmented Skin Lesions with Fusion and Analysis of Heterogeneous Data Based on a Multimodal Neural Network

Lyakhov

Lyakhova

Nagornov

2022

Cancers

View full text Add to dashboard Cite

show abstract

“…This can be explained by the fact that the model can learn some spurious correlations, causing interpretability methods to give exaggerated importance to those spurious regions highlighted on the produced saliency maps. In the same context, Wang et al [141] proposed a multimodal CNN for skin lesion diagnosis, which considered patient metadata and the skin lesion images. To analyze the contribution of each feature regarding the patient metadata, they adopted SHAP.…”

Section: Datasetmentioning

confidence: 99%

Explainable Deep Learning Methods in Medical Imaging Diagnosis: A Survey

Patrício¹,

Neves²,

Teixeira³

2022

Preprint

View full text Add to dashboard Cite

The remarkable success of deep learning has prompted interest in its application to medical diagnosis. Even tough state-ofthe-art deep learning models have achieved human-level accuracy on the classification of different types of medical data, these models are hardly adopted in clinical workflows, mainly due to their lack of interpretability. The black-box-ness of deep learning models has raised the need for devising strategies to explain the decision process of these models, leading to the creation of the topic of eXplainable Artificial Intelligence (XAI). In this context, we provide a thorough survey of XAI applied to medical diagnosis, including visual, textual, and example-based explanation methods. Moreover, this work reviews the existing medical imaging datasets and the existing metrics for evaluating the quality of the explanations . Complementary to most existing surveys, we include a performance comparison among a set of report generation-based methods. Finally, the major challenges in applying XAI to medical imaging are also discussed.CCS Concepts: • Applied computing → Health care information systems.

show abstract

“…3 Department of Information Science and Engineering, Northeastern University, Shen Yang, China. 4 Department of Information Science and Engineering, Northeastern University, Shen Yang, China. 5 Department of Information Science and Engineering, Northeastern University, Shen Yang, China.…”

Section: Appendix Acknowledgementsmentioning

confidence: 99%

“…The recent advancements made in deep learning have been applied to different medical fields in order to detect (at an earlier stage) or predict certain anomalies [3][4][5]. In the ophthalmological domain, digital fundus images analysis using deep learning methods have started to receive a lot of attention.…”

Section: Introductionmentioning

confidence: 99%

MulTiNet: Multimodal Neural Networks for Glaucoma Based on Transfer Learning

Han

Zhao

et al. 2021

Preprint

View full text Add to dashboard Cite

Background: Being one of the most serious causes of irreversible blindness, glaucoma has many subtypes and complex symptoms. In clinic, doctors usually need to use a variety of medical images for diagnosis. Optical Coherence Tomography (OCT), Visual Field (VF) , Fundus Photosexams (FP) and Ultrasonic BioMicroscope (UBM) are widely-used and complementary techniques for diagnosing glaucoma.Methods: At present, the field of intelligent diagnosis of glaucoma is limited by two major problems. One is the small number of data sets, and the other is the low diagnostic accuracy of Single-Modal Modal. In order to solve the above two problems, we have done the following work. First, we construct DualSY glaucoma multimodal data set. The four most important subtypes of glaucoma are discussed in this article which are Primary Open Angle Glaucoma (POAG), Primary Angle Closure Glaucoma (PACG), Primary Angle Closure Suspect (PACS) and Primary Angle Closure (PAC). Each patient in the DualSY data set contains more than five medical images, as shown in the figure 4.And DualSY are labeled with image-level multi-labels. Second, We propose a new Multi-Modal classification network for glaucoma, which is a multiclass classification model with various medical images of glaucoma patients and text information as input. The network structure consists of three main branches to deal with patient metadata, domain-based glaucoma features and medical images. Transfer learning method is introduced into this paper due to the small number of medical image data sets. The flowchart is shown in Figure 5.Result: Our method on glaucoma diagnosis outperforms state-of-the-art methods. A promising average result of overall accuracy (ACC) of 94.7% is obtained. Our data set outperformed most data sets in glaucoma diagnosis with an accuracy of 87.8%.Conclusions: The results suggest that medical images such as Heidelberg OCT and three-dimensional fundus photos used in this paper can better express the high-level information of glaucoma and our modal greatly improve the accuracy of glaucoma diagnosis. At the same time, this data set has great potential, and we continue to study this data.

show abstract

Interpretability-Based Multimodal Convolutional Neural Networks for Skin Lesion Diagnosis

Cited by 56 publications

References 42 publications

System for the Recognizing of Pigmented Skin Lesions with Fusion and Analysis of Heterogeneous Data Based on a Multimodal Neural Network

System for the Recognizing of Pigmented Skin Lesions with Fusion and Analysis of Heterogeneous Data Based on a Multimodal Neural Network

Explainable Deep Learning Methods in Medical Imaging Diagnosis: A Survey

MulTiNet: Multimodal Neural Networks for Glaucoma Based on Transfer Learning

Contact Info

Product

Resources

About