Discrepancy and Gradient-Guided Multi-modal Knowledge Distillation for Pathological Glioma Grading

Xing, Xiaohan; Chen, Zhe; Zhu, Meilu; Hou, Yuenan; Gao, Zhifan; Yuan, Yixuan

doi:10.1007/978-3-031-16443-9_61

Cited by 16 publications

(5 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hu et al [53] proposed to use knowledge distillation to transfer knowledge from a trained multi-modal network to a mono-modal one for medical image segmentation. Xing et al [54] proposed a discrepancy and gradient-guided knowledge distillation framework to transfer privileged knowledge from a multi-modal teacher network to a student network for pathological glioma grading. Yang et al [55] proposed an affinity-guided dense tumor-region knowledge distillation mechanism to align features for brain tumor segmentation with missing modalities.…”

Section: Knowledge Distillation On Medical Image Analysismentioning

confidence: 99%

A Transformer-Based Knowledge Distillation Network for Cortical Cataract Grading

Wang,

Xu,

Zheng

et al. 2024

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

Cortical cataract, a common type of cataract, is particularly difficult to be diagnosed automatically due to the complex features of the lesions. Recently, many methods based on edge detection or deep learning were proposed for automatic cataract grading. However, these methods suffer a large performance drop in cortical cataract grading due to the more complex cortical opacities and uncertain data. In this paper, we propose a novel Transformerbased Knowledge Distillation Network, called TKD-Net, for cortical cataract grading. To tackle the complex opacity problem, we first devise a zone decomposition strategy to extract more refined features and introduce special subscores to consider critical factors of clinical cortical opacity assessment (location, area, density) for comprehensive quantification. Next, we develop a multi-modal mixattention Transformer to efficiently fuse sub-scores and image modality for complex feature learning. However, obtaining the sub-score modality is a challenge in the clinic, which could cause the modality missing problem instead. To simultaneously alleviate the issues of modality missing and uncertain data, we further design a Transformer-based knowledge distillation method, which uses a teacher model with perfect data to guide a student model with modalitymissing and uncertain data. We conduct extensive experiments on a dataset of commonly-used slit-lamp images annotated by the LOCS III grading system to demonstrate that our TKD-Net outperforms state-of-the-art methods, as well as the effectiveness of its key components. Codes are available at https://github.com/wjh892521292/Cataract TKD-Net.

show abstract

Section: Knowledge Distillation On Medical Image Analysismentioning

confidence: 99%

A Transformer-Based Knowledge Distillation Network for Cortical Cataract Grading

Wang,

Xu,

Zheng

et al. 2024

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

show abstract

“…Transfer knowledge for missing modality robustness Recent studies (Zhou et al 2021b;Shen and Gao 2019;Wang et al 2021;Azad, Khosravi, and Merhof 2021) have introduced Knowledge Distillation (Gou et al 2021;Xing et al 2022) or co-training (Nigam and Ghani 2000) for improving missing-modality robustness. Despite the computational and memory cost of additional networks in their method, the choice of similarity measurements significantly impacts the final performance (Azad, Khosravi, and Merhof 2021).…”

Section: Related Workmentioning

confidence: 99%

“…(2) reconstruction-based modal-completion; (3) missingmodality robust architecture. For distillation or co-training approaches (Xing et al 2022;Wang et al 2021;Poklukar et al 2022), Although they marginally improve the missingmodality robustness, their methods are highly dependent on the choice of similarity measurements and require modalincomplete input to be filled with masking values (Wang et al 2021;Tsai et al 2019a), which may cause unexpected behavior in the models, leading to degraded performance (Shen and Gao 2019). Reconstruction-based methods (Khat-…”

Section: Introductionmentioning

confidence: 99%

Gradient-Guided Modality Decoupling for Missing-Modality Robustness

Wang,

Luo,

et al. 2024

AAAI

View full text Add to dashboard Cite

Multimodal learning with incomplete input data (missing modality) is very practical and challenging. In this work, we conduct an in-depth analysis of this challenge and find that modality dominance has a significant negative impact on the model training, greatly degrading the missing modality performance. Motivated by Grad-CAM, we introduce a novel indicator, gradients, to monitor and reduce modality dominance which widely exists in the missing-modality scenario. In aid of this indicator, we present a novel Gradient-guided Modality Decoupling (GMD) method to decouple the dependency on dominating modalities. Specifically, GMD removes the conflicted gradient components from different modalities to achieve this decoupling, significantly improving the performance. In addition, to flexibly handle modal-incomplete data, we design a parameter-efficient Dynamic Sharing (DS) framework which can adaptively switch on/off the network parameters based on whether one modality is available. We conduct extensive experiments on three popular multimodal benchmarks, including BraTS 2018 for medical segmentation, CMU-MOSI, and CMU-MOSEI for sentiment analysis. The results show that our method can significantly outperform the competitors, showing the effectiveness of the proposed solutions. Our code is released here: https://github.com/HaoWang420/Gradient-guided-Modality-Decoupling.

show abstract

“…Although coordinated representations have traditionally tended to be more challenging to implement, the convenience of neural network architectural and loss adjustments have resulted in increased traction in publications embodying coordinated representations (Xing et al, 2022;Chauhan et al, 2020;Radford et al, 2021;Bhalodia et al, 2021). One of the most notable of these in recent AI approaches is OpenAI's Contrastive Language-Image Pre-Training (CLIP) model, which forms representations for OpenAI's DALL•E 2 (Radford et al, 2021;Ramesh et al, 2022) and uses a contrastive-learning approach to shape both image embeddings of entire images to match text embeddings of entire captions describing those images.…”

Section: Tablementioning

confidence: 99%

“…In privileged models based on traditional approaches (before deep neural networks), privileged information can be embedded in the model either through an alteration of allowable error ("slack variables" from SVM+) (Vapnik and Vashist, 2009), or through decision trees constructed with non-privileged features to mimic the discriminative ability of privileged features (Random Forest+) (Warner et al, 2022;Moradi et al, 2016). In a deep learning model, privileged learning is often achieved through the use of additional loss functions which attempt to constrain latent and output vectors from the non-privileged modality to mimic those from the combined privileged and non-privileged models Xing et al, 2022). For example, in Chauhan et al (2020), encoders for each modality are compared and cross entropy loss is calculated for each modality separately.…”

Section: Privileged Learningmentioning

confidence: 99%

Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

Warner,

Lee,

Hsu

et al. 2024

Int J Comput Vis

View full text Add to dashboard Cite

Machine learning (ML) applications in medical artificial intelligence (AI) systems have shifted from traditional and statistical methods to increasing application of deep learning models. This survey navigates the current landscape of multimodal ML, focusing on its profound impact on medical image analysis and clinical decision support systems. Emphasizing challenges and innovations in addressing multimodal representation, fusion, translation, alignment, and co-learning, the paper explores the transformative potential of multimodal models for clinical predictions. It also highlights the need for principled assessments and practical implementation of such models, bringing attention to the dynamics between decision support systems and healthcare providers and personnel. Despite advancements, challenges such as data biases and the scarcity of “big data” in many biomedical domains persist. We conclude with a discussion on principled innovation and collaborative efforts to further the mission of seamless integration of multimodal ML models into biomedical practice.

show abstract

Discrepancy and Gradient-Guided Multi-modal Knowledge Distillation for Pathological Glioma Grading

Cited by 16 publications

References 24 publications

A Transformer-Based Knowledge Distillation Network for Cortical Cataract Grading

A Transformer-Based Knowledge Distillation Network for Cortical Cataract Grading

Gradient-Guided Modality Decoupling for Missing-Modality Robustness

Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

Contact Info

Product

Resources

About