Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation

Ji, Mingi; Shin, Seungjae; Hwang, Seunghyun; Park, Gibeom; Moon, Il‐Chul

doi:10.1109/cvpr46437.2021.01052

Cited by 119 publications

(58 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Knowledge distillation is first proposed by Hinton (Hinton et al, 2015 ) to compress the model by transferring the knowledge from a cumbersome teacher network to a compact student network. Recently, knowledge distillation is also applied to the model enhancement through improved learning strategy [including self-learning (Ji et al, 2021 ; Zheng and Peng, 2022 ) and mutual learning (Li et al, 2021 )]. For example, Hong et al ( 2020 ) applies knowledge distillation to heterogeneous task imitation and guides the student network training using the features extracted from the image reconstruction task.…”

Section: Related Workmentioning

confidence: 99%

Physical-model guided self-distillation network for single image dehazing

Lan

Cui

et al. 2022

Front. Neurorobot.

View full text Add to dashboard Cite

MotivationImage dehazing, as a key prerequisite of high-level computer vision tasks, has gained extensive attention in recent years. Traditional model-based methods acquire dehazed images via the atmospheric scattering model, which dehazed favorably but often causes artifacts due to the error of parameter estimation. By contrast, recent model-free methods directly restore dehazed images by building an end-to-end network, which achieves better color fidelity. To improve the dehazing effect, we combine the complementary merits of these two categories and propose a physical-model guided self-distillation network for single image dehazing named PMGSDN.Proposed methodFirst, we propose a novel attention guided feature extraction block (AGFEB) and build a deep feature extraction network by it. Second, we propose three early-exit branches and embed the dark channel prior information to the network to merge the merits of model-based methods and model-free methods, and then we adopt self-distillation to transfer the features from the deeper layers (perform as teacher) to shallow early-exit branches (perform as student) to improve the dehazing effect.ResultsFor I-HAZE and O-HAZE datasets, better than the other methods, the proposed method achieves the best values of PSNR and SSIM being 17.41dB, 0.813, 18.48dB, and 0.802. Moreover, for real-world images, the proposed method also obtains high quality dehazed results.ConclusionExperimental results on both synthetic and real-world images demonstrate that the proposed PMGSDN can effectively dehaze images, resulting in dehazed results with clear textures and good color fidelity.

show abstract

Section: Related Workmentioning

confidence: 99%

Physical-model guided self-distillation network for single image dehazing

Lan

Cui

et al. 2022

Front. Neurorobot.

View full text Add to dashboard Cite

show abstract

“…(1) KD first requires training a large DNN as the teacher; (2) when training the student, KD needs to process each sample twice in each iteration, once by the teacher and once by the student. To reduce the cost of training a large teacher, many self-distillation approaches including but not limited to (Xu and Liu 2019;Zhang et al 2019;Yang et al 2019b,a;Furlanello et al 2018;Bagherinezhad et al 2018;Yun et al 2020;Deng and Zhang 2021c;Ji et al 2021) have been proposed. Zhang et al (2019) and Ji et al (2021) add additional layers or parameters to a DNN to generate soft labels, which improves the performance but introduces large computation and memory cost.…”

Section: Related Workmentioning

confidence: 99%

“…To reduce the cost of training a large teacher, many self-distillation approaches including but not limited to (Xu and Liu 2019;Zhang et al 2019;Yang et al 2019b,a;Furlanello et al 2018;Bagherinezhad et al 2018;Yun et al 2020;Deng and Zhang 2021c;Ji et al 2021) have been proposed. Zhang et al (2019) and Ji et al (2021) add additional layers or parameters to a DNN to generate soft labels, which improves the performance but introduces large computation and memory cost. Born-again networks (Furlanello et al 2018;Yang et al 2019a) and label-refine networks (Bagherinezhad et al 2018) train a DNN for many generations and the network in the (i − 1)th generation is used as the teacher to train the network in the ith generation.…”

Section: Related Workmentioning

confidence: 99%

Reducing Flipping Errors in Deep Neural Networks

Deng

Xiao

Long

et al. 2022

AAAI

View full text Add to dashboard Cite

Deep neural networks (DNNs) have been widely applied in various domains in artificial intelligence including computer vision and natural language processing. A DNN is typically trained for many epochs and then a validation dataset is used to select the DNN in an epoch (we simply call this epoch ``the last epoch") as the final model for making predictions on unseen samples, while it usually cannot achieve a perfect accuracy on unseen samples. An interesting question is ``how many test (unseen) samples that a DNN misclassifies in the last epoch were ever correctly classified by the DNN before the last epoch?". In this paper, we empirically study this question and find on several benchmark datasets that the vast majority of the misclassified samples in the last epoch were ever classified correctly before the last epoch, which means that the predictions for these samples were flipped from ``correct" to ``wrong". Motivated by this observation, we propose to restrict the behavior changes of a DNN on the correctly-classified samples so that the correct local boundaries can be maintained and the flipping error on unseen samples can be largely reduced. Extensive experiments on different benchmark datasets with different modern network architectures demonstrate that the proposed flipping error reduction (FER) approach can substantially improve the generalization, the robustness, and the transferability of DNNs without introducing any additional network parameters or inference cost, only with a negligible training overhead.

show abstract

“…There are two categories of self-distillation techniques, namely the auxiliary parameter methods [2,8,13,30,33] and contrastive sample methods [14,27,29,31]. Auxiliary model methods exploit additional branches to get extra predictions besides the main-branch prediction for soft label supervision at the cost of more parameters overhead.…”

Section: Self Distillationmentioning

confidence: 99%

“…Self distillation simplifies the two-stage knowledge distillation framework by distilling knowledge from itself instead of from the pretrained teacher, and still [7,12]. b) Auxiliary parameters [8,13,33]. c) Progressive distillation with memory bank storing past predictions of the entire dataset [2,14,31].…”

Section: Introductionmentioning

confidence: 99%

Efficient One Pass Self-distillation with Zipf's Label Smoothing

Liang¹,

Li²,

Bing³

et al. 2022

Preprint

View full text Add to dashboard Cite

Self-distillation exploits non-uniform soft supervision from itself during training and improves performance without any runtime cost. However, the overhead during training is often overlooked, and yet reducing time and memory overhead during training is increasingly important in the giant models' era. This paper proposes an efficient selfdistillation method named Zipf's Label Smoothing (Zipf's LS), which uses the on-the-fly prediction of a network to generate soft supervision that conforms to Zipf distribution without using any contrastive samples or auxiliary parameters. Our idea comes from an empirical observation that when the network is duly trained the output values of a network's final softmax layer, after sorting by the magnitude and averaged across samples, should follow a distribution reminiscent to Zipf's Law in the word frequency statistics of natural languages. By enforcing this property on the sample level and throughout the whole training period, we find that the prediction accuracy can be greatly improved. Using ResNet50 on the INAT21 fine-grained classification dataset, our technique achieves +3.61% accuracy gain compared to the vanilla baseline, and 0.88% more gain against the previous label smoothing or selfdistillation strategies. The implementation is publicly available at https: //github.com/megvii-research/zipfls.

show abstract

Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation

Cited by 119 publications

References 23 publications

Physical-model guided self-distillation network for single image dehazing

Physical-model guided self-distillation network for single image dehazing

Reducing Flipping Errors in Deep Neural Networks

Efficient One Pass Self-distillation with Zipf's Label Smoothing

Contact Info

Product

Resources

About