Feature Fusion for Online Mutual Knowledge Distillation

Kim, Jangho; Hyun, Minsung; Chung, Inseop; Kwak, Nojun

doi:10.1109/icpr48806.2021.9412615

Cited by 56 publications

(34 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, there are many new online distillation methods that train a teacher and a student simultaneously during knowledge transfer. Collaborative learning is the one used most often [25,29,32,8,9,7], where the teacher and the student as peer networks collaboratively teach and learn from each other, and the peer network architectures can be different. In particular, Zhang et al [25] proposed a deep mutual learning method (DML) for online distillation using the response-based knowledge.…”

Section: Collaborative Learningmentioning

confidence: 99%

“…Unlike the ensemble of peer networks, the advantage of a mutual knowledge distillation method is that it can fuse features of peer networks to collaboratively learn a powerful classifier [32]. However, the knowledge distilled by those online mutual distillation methods is limited to the response-based knowledge from individual instance features.…”

Section: Collaborative Learningmentioning

confidence: 99%

“…As pointed out in [1], there are three kinds of knowledge, i.e., the response-based, the feature-based and the relation-based knowledge. Generally, most of KD methods distill the response-based knowledge (e.g., soft logits of the output layer) from a large teacher network and transfer it to a small student [20,32,22]. To overcome the limitation of knowledge from the output layer of teacher, the featurebased knowledge from the middle layers of teacher is also used to train the student [21,18,19].…”

Section: Introductionmentioning

confidence: 99%

“…Thus, finding a proper teacher-student architecture in offline distillation is challenging. In contrast, online distillation provides a one-phase end-to-end training scheme via teacher-student collaborative learning on a peer-network architecture instead of a fixed one [25,33,36,32,28,12].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Collaborative Teacher-Student Learning via Multiple Knowledge Transfer

Sun¹,

Gou²,

Yu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Knowledge distillation (KD), as an efficient and effective model compression technique, has been receiving considerable attention in deep learning. The key to its success is to transfer knowledge from a large teacher network to a small student one. However, most of the existing knowledge distillation methods consider only one type of knowledge learned from either instance features or instance relations via a specific distillation strategy in teacher-student learning. There are few works that explore the idea of transferring different types of knowledge with different distillation strategies in a unified framework. Moreover, the frequently used offline distillation suffers from a limited learning capacity due to the fixed teacher-student architecture. In this paper we propose a collaborative teacher-student learning via multiple knowledge transfer (CTSL-MKT) that prompts both self-learning and collaborative learning. It allows multiple students learn knowledge from both individual instances and instance relations in a collaborative way. While learning from themselves with self-distillation, they can also guide each other via online distillation. The experiments and ablation studies on four image datasets demonstrate that the proposed CTSL-MKT significantly outperforms the state-of-the-art KD methods.

show abstract

Section: Collaborative Learningmentioning

confidence: 99%

Section: Collaborative Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Collaborative Teacher-Student Learning via Multiple Knowledge Transfer

Sun¹,

Gou²,

Yu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Teacher network transfers its knowledge to student network to enhance the performance of student network. Feature maps [8,9,10] and logits of a network [11,12] are widely used as knowledge. Model compression has actively been studied mainly on computer vision tasks.…”

Section: Introductionmentioning

confidence: 99%

PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation

Kim¹,

Chang²,

Kwak³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

As edge devices become prevalent, deploying Deep Neural Networks (DNN) on edge devices has become a critical issue. However, DNN requires a high computational resource which is rarely available for edge devices. To handle this, we propose a novel model compression method for the devices with limited computational resources, called PQK consisting of pruning, quantization, and knowledge distillation (KD) processes. Unlike traditional pruning and KD, PQK makes use of unimportant weights pruned in the pruning process to make a teacher network for training a better student network without pre-training the teacher model. PQK has two phases. Phase 1 exploits iterative pruning and quantization-aware training to make a lightweight and power-efficient model. In phase 2, we make a teacher network by adding unimportant weights unused in phase 1 to a pruned network. By using this teacher network, we train the pruned network as a student network. In doing so, we do not need a pre-trained teacher network for the KD framework because the teacher and the student networks coexist within the same network (See Fig. 1). We apply our method to the recognition model and verify the effectiveness of PQK on keyword spotting (KWS) and image recognition.

show abstract