Improved Knowledge Distillation via Teacher Assistant

Mirzadeh, Seyed Iman; Farajtabar, Mehrdad; Li, Ang; Levine, Nir; Matsukawa, Akihiro; Ghasemzadeh, Hassan

doi:10.1609/aaai.v34i04.5963

Cited by 700 publications

(292 citation statements)

References 10 publications

(16 reference statements)

Supporting

Mentioning

289

Contrasting

Order By: Relevance

“…When it comes to the effects of knowledge definition, the structural differences between the networks are very important. [25] also finds that networks with similar structures are easier to transfer knowledge. Therefore, in this paper, the knowledge distillation between the sub-networks of the original network is used to minimize the structural differences.…”

Section: Knowledge Distillationmentioning

confidence: 90%

Knowledge from the original network: restore a better pruned network with knowledge distillation

Chen

et al. 2021

Complex Intell. Syst.

View full text Add to dashboard Cite

To deploy deep neural networks to edge devices with limited computation and storage costs, model compression is necessary for the application of deep learning. Pruning, as a traditional way of model compression, seeks to reduce the parameters of model weights. However, when a deep neural network is pruned, the accuracy of the network will significantly decrease. The traditional way to decrease the accuracy loss is fine-tuning. When over many parameters are pruned, the pruned network’s capacity is reduced heavily and cannot recover to high accuracy. In this paper, we apply the knowledge distillation strategy to abate the accuracy loss of pruned models. The original network of the pruned network was used as the teacher network, aiming to transfer the dark knowledge from the original network to the pruned sub-network. We have applied three mainstream knowledge distillation methods: response-based knowledge, feature-based knowledge, and relation-based knowledge (Gou et al. in Knowledge distillation: a survey. arXiv:200605525, 2020), and compare the result to the traditional fine-tuning method with grand-truth labels. Experiments have been done on the CIFAR100 dataset with several deep convolution neural network. Results show that the pruned network recovered by knowledge distillation with its original network performs better accuracy than it recovered by fine-tuning with sample labels. It has also been validated in this paper that the original network as the teacher performs better than differently structured networks with same accuracy as the teacher.

show abstract

Section: Knowledge Distillationmentioning

confidence: 90%

Knowledge from the original network: restore a better pruned network with knowledge distillation

Chen

et al. 2021

Complex Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…To alleviate the capacity mismatching problem, Ref. [ 35 ] introduces multi-step KD, which uses an intermediate-sized model (teacher assistant) to bridge the gap between the student and teacher. Route Constrained Optimization (RCO) [ 36 ] supervises the student model with some anchor points selected from the route in parameter space that the teacher pass by, instead of the converged teacher model.…”

Section: Related Workmentioning

confidence: 99%

Memory-Replay Knowledge Distillation

Wang

Zhang

2021

Sensors

View full text Add to dashboard Cite

Knowledge Distillation (KD), which transfers the knowledge from a teacher to a student network by penalizing their Kullback–Leibler (KL) divergence, is a widely used tool for Deep Neural Network (DNN) compression in intelligent sensor systems. Traditional KD uses pre-trained teacher, while self-KD distills its own knowledge to achieve better performance. The role of the teacher in self-KD is usually played by multi-branch peers or the identical sample with different augmentation. However, the mentioned self-KD methods above have their limitation for widespread use. The former needs to redesign the DNN for different tasks, and the latter relies on the effectiveness of the augmentation method. To avoid the limitation above, we propose a new self-KD method, Memory-replay Knowledge Distillation (MrKD), that uses the historical models as teachers. Firstly, we propose a novel self-KD training method that penalizes the KD loss between the current model’s output distributions and its backup outputs on the training trajectory. This strategy can regularize the model with its historical output distribution space to stabilize the learning. Secondly, a simple Fully Connected Network (FCN) is applied to ensemble the historical teacher’s output for a better guidance. Finally, to ensure the teacher outputs offer the right class as ground truth, we correct the teacher logit output by the Knowledge Adjustment (KA) method. Experiments on the image (dataset CIFAR-100, CIFAR-10, and CINIC-10) and audio (dataset DCASE) classification tasks show that MrKD improves single model training and working efficiently across different datasets. In contrast to the existing fancy self-KD methods with various external knowledge, the effectiveness of MrKD sheds light on the usually abandoned historical models during the training trajectory.

show abstract

“…In 2019, Mirzadeh et al [53] demonstrated that any desired teacher network is only capable of distilling knowledge to a student model with a specific threshold of parameters. If the size of the model is less than that threshold, the KD procedure may not be compelling.…”

Section: B Multi-teacher Kdmentioning

confidence: 99%

Robust Semantic Segmentation With Multi-Teacher Knowledge Distillation

Amirkhani¹,

Khosravian

Masih‐Tehrani

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Recent studies have recently exploited knowledge distillation (KD) technique to address timeconsuming annotation task in semantic segmentation, through which one teacher trained on a single dataset could be leveraged for annotating unlabeled data. However, in this context, knowledge capacity is restricted, and knowledge variety is rare in different conditions, such as cross-model KD, in which the single teacher KD prohibits the student model from distilling information using cross-domain context. To fix this concern, we have looked into learning a lightweight student from a group of teachers. To be more specific, we train five distinct lightweight convolutional neural networks (CNNs) for semantic segmentation on several datasets. Several state-of-the-art augmentation transformations have also been utilized in our training phase. The impacts of such training scenarios are then assessed in terms of student robustness and accuracy. As the main contribution of this paper, our proposed multi-teacher KD paradigm endows the student with the ability to amalgamate and capture a variety of knowledge illustrations from different sources. Results demonstrated that our method outperforms the existing studies on both clean and corrupted data in the semantic segmentation task while benefiting from our proposed score weight system. Experiments validate that our multi-teacher framework results in an improvement of 9% up to 32.18% compared to the singleteacher paradigm. Moreover, it is demonstrated that our paradigm surpasses previous supervised real-time studies in the semantic segmentation challenge.

show abstract

Improved Knowledge Distillation via Teacher Assistant

Cited by 700 publications

References 10 publications

Knowledge from the original network: restore a better pruned network with knowledge distillation

Knowledge from the original network: restore a better pruned network with knowledge distillation

Memory-Replay Knowledge Distillation

Robust Semantic Segmentation With Multi-Teacher Knowledge Distillation

Contact Info

Product

Resources

About