Memory-Replay Knowledge Distillation

Abstract

Knowledge Distillation (KD), which transfers the knowledge from a teacher to a student network by penalizing their Kullback–Leibler (KL) divergence, is a widely used tool for Deep Neural Network (DNN) compression in intelligent sensor systems. Traditional KD uses pre-trained teacher, while self-KD distills its own knowledge to achieve better performance. The role of the teacher in self-KD is usually played by multi-branch peers or the identical sample with different augmentation. However, the mentioned self-KD methods above have their limitation for widespread use. The former needs to redesign the DNN for different tasks, and the latter relies on the effectiveness of the augmentation method. To avoid the limitation above, we propose a new self-KD method, Memory-replay Knowledge Distillation (MrKD), that uses the historical models as teachers. Firstly, we propose a novel self-KD training method that penalizes the KD loss between the current model’s output distributions and its backup outputs on the training trajectory. This strategy can regularize the model with its historical output distribution space to stabilize the learning. Secondly, a simple Fully Connected Network (FCN) is applied to ensemble the historical teacher’s output for a better guidance. Finally, to ensure the teacher outputs offer the right class as ground truth, we correct the teacher logit output by the Knowledge Adjustment (KA) method. Experiments on the image (dataset CIFAR-100, CIFAR-10, and CINIC-10) and audio (dataset DCASE) classification tasks show that MrKD improves single model training and working efficiently across different datasets. In contrast to the existing fancy self-KD methods with various external knowledge, the effectiveness of MrKD sheds light on the usually abandoned historical models during the training trajectory.

Adaptive Aggregation Networks for Class-Incremental Learning

“…The replay-based methods focusing on memory optimization [28,45] are closely related to our work. [28] proposed a bilevel optimization framework to distill the current new class data into exemplars before discarding them.…”

“…Replay-based methods assume there is a clear memory budget allowing a handful of old-class exemplars in the memory. Exemplars can be used to re-train the model in each new phase [13,18,27,28,37,45,46]. This re-training usually contains two steps: one step of training the model on all new class data and old class exemplars, and one step of finetuning the model with a balanced subset (i.e., using an equal number of samples per class) [13,18,26,27,48].…”

“…It aims to improve the quality of exemplars without increasing the quantity. Another work [45] aimed to trade-off between the quality and quantity of exemplars by image compression using the JPEG algorithm, i.e., each exemplar is uniformly downsampled. Ours differs from these two works in three aspects.…”

“…(b) Mnemonics [28] distills all training samples into few-shot exemplars without increasing their quantity. (c) MRDC [45] compresses each exemplar uniformly into a low-resolution image using JPEG [43]. (d) Our approach based on the proposed class-incremental masking (CIM) downsamples only non-discriminative pixels in the image.…”

“…Liu et al [28] tried to mitigate this problem by parameterizing and distilling the exem-plars, without increasing the number of them (Figure 1b). Wang et al [45] traded off between the quality and quantity of exemplars by uniformly compressing exemplar images with JPEG [43] (Figure 1c). As shown in Figure 1d, our approach is also based on image compression.…”