Efficient Medical Image Segmentation Based on Knowledge Distillation

Qin, Dian; Bu, Jiajun; Liu, Zhe; Shen, Xipeng; Zhou, Sheng; Gu, Jingjun; Wang, Zhihua; Wu, Lei; Huifen, Dai

doi:10.1109/tmi.2021.3098703

Cited by 105 publications

(36 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The music signal feature extraction here is based on the steady-state signal. Therefore, before the music signal feature extraction, it is generally necessary to divide the frame ( Qin et al, 2021 ). In order to ensure that the information between the two signals during a seamless transition is not lost, it is necessary to ensure that the two signals have overlapping parts.…”

Section: Methodsmentioning

confidence: 99%

Construction of Intelligent Recognition and Learning Education Platform of National Music Genre Under Deep Learning

2022

Front. Psychol.

View full text Add to dashboard Cite

In order to study the application of the deep learning (DL) method in music genre recognition, this study introduces the music feature extraction method and the deep belief network (DBN) in DL and proposes the parameter extraction feature and the recognition classification method of an ethnic music genre based on the DBN with five kinds of ethnic musical instruments as the experimental objects. A national musical instrument recognition and classification network structure based on the DBN is proposed. On this basis, a music library classification retrieval learning platform has been established and tested. The results show that, when the DBN only contains one hidden layer and the number of neural nodes in the hidden layer is 117, the basic convergence accuracy is approximately 98%. The first hidden layer has the greatest impact on the prediction results. When the input sample feature size is one-third of the number of nodes in the first hidden layer, the network performance is basically convergent. The DBN is the best way for softmax to identify and classify national musical instruments, and the accuracy rate is 99.2%. Therefore, the proposed DL algorithm performs better in identifying music genres.

show abstract

Section: Methodsmentioning

confidence: 99%

Construction of Intelligent Recognition and Learning Education Platform of National Music Genre Under Deep Learning

2022

Front. Psychol.

View full text Add to dashboard Cite

show abstract

“…Wen et al [20] propose a boundary-guided knowledge distillation method which assists the student network to align organ boundary features in teacher network. Qin et al [21] design a region affinity distillation method, and integrate the importance map for efficient liver tumor segmentation. However, these methods mostly concentrate on distilling the knowledge from certain layers, while do not notice the variations of pathological or organic semantic-aware representations from different layers.…”

Section: B Knowledge Distillationmentioning

confidence: 99%

“…Wen et al [20] propose a boundary-guided distillation method for organ segmentation. Qin et al [21] design a region affinity distillation method for efficient liver tumor segmentation. Tran et al [22] propose a new Light-weight Deformable Registration network that introduce a new adversarial learning with distilling knowledge.…”

Section: Introductionmentioning

confidence: 99%

Graph Flow: Cross-layer Graph Flow Distillation for Dual Efficient Medical Image Segmentation

Zou¹,

Sun²

2022

Preprint

View full text Add to dashboard Cite

With the development of deep convolutional neural networks, medical image segmentation has achieved a series of breakthroughs in recent years. However, the higher-performance convolutional neural networks always mean numerous parameters and high computation costs, which will hinder the applications in clinical scenarios. Meanwhile, the scarceness of large-scale annotated medical image datasets further impedes the application of high-performance networks. To tackle these problems, we propose Graph Flow, a comprehensive knowledge distillation framework, for both network-efficiency and annotation-efficiency medical image segmentation. Specifically, our core Graph Flow Distillation transfer the essence of cross-layer variations from a well-trained cumbersome teacher network to a non-trained compact student network. In addition, an unsupervised Paraphraser Module is designed to purify the knowledge of the teacher network, which is also beneficial for the stabilization of training procedure. Furthermore, we build a unified distillation framework by integrating the adversarial distillation and the vanilla logits distillation, which can further refine the final predictions of the compact network. Extensive experiments conducted on Gastric Cancer Segmentation Dataset and Synapse Multi-organ Segmentation Dataset demonstrate the prominent ability of our method which achieves state-of-the-art performance on these differentmodality and multi-category medical image datasets. Moreover, we demonstrate the effectiveness of our Graph Flow through a new semi-supervised paradigm for dual efficient medical image segmentation. Our code will be available at Graph Flow .

show abstract

“…Self-distillation explores the potential of knowledge distillation from a new perspective [31]. Similarly, self-distillation strategies can be summarized as extracting the attention map of the current layers and then transferring the knowledge to the previous layer [32].…”

Section: Self-distillation Learningmentioning

confidence: 99%

MISSU: 3D Medical Image Segmentation via Self-distilling TransUNet

Wang¹,

Lin²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

U-Nets have achieved tremendous success in medical image segmentation. Nevertheless, it may suffer limitations in global (long-range) contextual interactions and edge-detail preservation. In contrast, Transformer has an excellent ability to capture long-range dependencies by leveraging the self-attention mechanism into the encoder. Although Transformer was born to model the long-range dependency on the extracted feature maps, it still suffers from extreme computational and spatial complexities in processing high-resolution 3D feature maps. This motivates us to design the efficiently Transformer-based UNet model and study the feasibility of Transformer-based network architectures for medical image segmentation tasks. To this end, we propose to self-distill a Transformer-based UNet for medical image segmentation, which simultaneously learns global semantic information and local spatial-detailed features. Meanwhile, a local multi-scale fusion block is first proposed to refine finegrained details from the skipped connections in the encoder by the main CNN stem through self-distillation, only computed during training and removed at inference with minimal overhead. Extensive experiments on BraTS 2019 and CHAOS datasets show that our MISSU achieves the best performance over previous state-of-the-art methods. Code and models are available at https://github.com/wangn123/MISSU.git

show abstract

Efficient Medical Image Segmentation Based on Knowledge Distillation

Cited by 105 publications

References 38 publications

Construction of Intelligent Recognition and Learning Education Platform of National Music Genre Under Deep Learning

Construction of Intelligent Recognition and Learning Education Platform of National Music Genre Under Deep Learning

Graph Flow: Cross-layer Graph Flow Distillation for Dual Efficient Medical Image Segmentation

MISSU: 3D Medical Image Segmentation via Self-distilling TransUNet

Contact Info

Product

Resources

About