Learn From the Past: Experience Ensemble Knowledge Distillation

Abstract

Traditional knowledge distillation transfers "dark knowledge" of a pre-trained teacher network to a student network, and ignores the knowledge in the training process of the teacher, which we call teacher's experience. However, in realistic educational scenarios, learning experience is often more important than learning results. In this work, we propose a novel knowledge distillation method by integrating the teacher's experience for knowledge transfer, named experience ensemble knowledge distillation (EEKD). We save a moderate number of intermediate models from the training process of the teacher model uniformly, and then integrate the knowledge of these intermediate models by ensemble technique. A self-attention module is used to adaptively assign weights to different intermediate models in the process of knowledge transfer. Three principles of constructing EEKD on the quality, weights and number of intermediate models are explored. A surprising conclusion is found that strong ensemble teachers do not necessarily produce strong students. The experimental results on CIFAR-100 and ImageNet show that EEKD outperforms the mainstream knowledge distillation methods and achieves the state-of-theart. In particular, EEKD even surpasses the standard ensemble distillation on the premise of saving training cost.

A Survey of Historical Learning: Learning Models with Learning History

“…These past elements are recorded and will be replaced based on the firstin, first-out principle when the memory bank or queue reaches its capacity. The iterative teacher model maintains historical DEs [36], [53]- [58]. Feng et al [36] perform teacher learning by incorporating multiple historical model checkpoints in a parallel manner.…”

“…(2) Simple Moving Average (SMA) is a type of arithmetic average that is calculated by adding recent data and dividing the sum by the number of time periods. The works in [14], [25]- [33], [57], [58], [135] all use SMA to obtain the average multiple checkpoints, predictions or gradients of features during the learning process. SMA has been proven to result in wider optima and better generalization.…”

“…To achieve the best ensemble effect at the lowest cost, some work investigates how various hyperparameters [167], [168], architectures [169], [170], strategies [168], [171], [172], and distribution shift [173], [174] affect ensemble models. Ensemble learning primarily makes use of historical information by ensembling the historical results [15], [16] and parameters [57], [58] to enhance the model's performance during training and inference. However, two exceptional cases cannot be considered as part of historical learning.…”