Improved Knowledge Distillation via Teacher Assistant

Abstract

Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too large to be deployed on edge devices like smartphones or embedded sensor nodes. There have been efforts to compress these networks, and a popular method is knowledge distillation, where a large (teacher) pre-trained network is used to train a smaller (student) network. However, in this paper, we show that the student network performance degrades when the gap between student and teacher is large. Given a fixed student network, one cannot employ an arbitrarily large teacher, or in other words, a teacher can effectively transfer its knowledge to students up to a certain size, not smaller. To alleviate this shortcoming, we introduce multi-step knowledge distillation, which employs an intermediate-sized network (teacher assistant) to bridge the gap between the student and the teacher. Moreover, we study the effect of teacher assistant size and extend the framework to multi-step distillation. Theoretical analysis and extensive experiments on CIFAR-10,100 and ImageNet datasets and on CNN and ResNet architectures substantiate the effectiveness of our proposed approach.

Compacting Deep Neural Networks for Internet of Things: Methods and Applications

“…Yu et al [100] proposed two new loss functions to simulate the communication between the deep teacher network and the small student network: one is based on the absolute teachers, the other is based on the relative teacher network. Mirzadeh et al [101] introduced multi-step knowledge extraction technology and used a medium-sized network (teacher assistant) to fill the gap between students and teachers.…”

“…Knowledge from Details Hinton et al [97] Logits Cross Entropy Huang et al [99] Logits Cross Entropy and maximum mean discrepancy Yu et al [100] Logits Hints and attention Mirzadeh et al [101] Logits Use teacher assistant Romero et al [102] Intermediate layers MSEloss in a certain middle layer Yim et al [103] Intermediate layers Gram matrix loss in multiple middle layers Zagoruyko et al [104] Intermediate layers Attention transfer loss in multiple middle layers Zhang et al [105] Intermediate layers Adaptive selection a middle layer Peng et al [106] Mutual information Correlation between multiple instances Crowley et al [107] Self structures With the same structure, use cheap convolution blocks Park et al [108] Structured knowledge Use a relational potential function to transfers the information Lopez-Paz et al [109] Privileged information Use pair-wise distillation and holistic distillation between two neural networks and proposed an information theory framework for knowledge transfer. Tung et al [112] proposed a new form of KD loss, which was inspired by a similar input pattern in a well-trained network.…”

Be Your Own Best Competitor! Multi-Branched Adversarial Knowledge Transfer

“…In [33], the capacity gap between the large teacher model and the student has been investigated. It shows that the relationships between the architecture of the teacher and student model are very important.…”

Designing Interpretable Approximations to Deep Reinforcement Learning

“…Deep neural network (DNN)-driven algorithms now stand as the state of the art in a variety of domains, from perceptual tasks such as computer vision, speech and language processing to, more recently, control tasks such as robotics (Mirzadeh et al 2019), (Bastani, Pu, and Solar-Lezama 2018). Nevertheless, there is often reason to avoid direct use of DNNs.…”

“…For example, the training from scratch or hyperparameter tuning of such networks can be prohibitively expensive or time consuming (Schmitt et al 2018). For some applications, the size or complexity of such DNNs precludes their use in real time, or employment in edge devices with limited processing resources (Chen et al 2017), (Mirzadeh et al 2019). In other areas such as flight control or self driving cars, DNNs are sidelined (at least for mass deployment) by their opaqueness or lack of decision making interpretability (Bastani, Kim, and Bastani 2017), (Hind et al 2019).…”