Online Ensemble Model Compression Using Knowledge Distillation

Abstract

This paper presents a novel knowledge distillation based model compression framework consisting of a student ensemble. It enables distillation of simultaneously learnt ensemble knowledge onto each of the compressed student models. Each model learns unique representations from the data distribution due to its distinct architecture. This helps the ensemble generalize better by combining every model's knowledge. The distilled students and ensemble teacher are trained simultaneously without requiring any pretrained weights. Moreover, our proposed method can deliver multi-compressed students with single training, which is efficient and flexible for different scenarios. We provide comprehensive experiments using state-of-the-art classification models to validate our framework's effectiveness. Notably, using our framework a 97% compressed ResNet110 student model managed to produce a 10.64% relative accuracy gain over its individual baseline training on CIFAR100 dataset. Similarly a 95% compressed DenseNet-BC (k=12) model managed a 8.17% relative accuracy gain.

Knowledge distillation in deep learning and its applications

“…To demonstrate the usage of distillation metric, we use the results reported in Walawalkar, Shen & Savvides (2020) using CIFAR100 dataset Krizhevsky (2009) and the Resnet44 architecture He et al (2016) . In their experiment, they trained four student models having relative sizes of 62.84%, 35.36%, 15.25% and 3.74% as compared to the teacher model.…”

“… Walawalkar, Shen & Savvides (2020) proposed to train an ensemble of models that is broken down into four blocks, where all models share the first block of layers. The first model in the ensemble is considered the teacher (termed pseudo teacher in the paper).…”

“…After training, any model in the ensemble can be selected to be deployed or the whole ensemble with the fusion classifier can be deployed in case of lenient hardware constraints. Walawalkar, Shen & Savvides (2020) proposed to train an ensemble of models that is broken down into four blocks, where all models share the first block of layers. The first model in the ensemble is considered the teacher (termed pseudo teacher in the paper).…”

Collaborative Teacher-Student Learning via Multiple Knowledge Transfer

“…Thus, finding a proper teacher-student architecture in offline distillation is challenging. In contrast, online distillation provides a one-phase end-to-end training scheme via teacher-student collaborative learning on a peer-network architecture instead of a fixed one [25,33,36,32,28,12].…”

Dynamic sparsity and model feature learning enhanced training for convolutional neural network-pruning

“…受深度互学习 [17] 和在线集成模型压缩 [18] 方法启 发, 为了使压缩模型中每一个压缩子模型之间互相 增强学习, 本文将压缩子模型(去除需要学习更新权 重的模型)的类别输出平均值作为监督信息, 同样采 用该输出平均值的 Softmax 概率与需要更新学习权重 的压缩子模型输出预测值 Softmax 概率的 KL 散度作 为损失函数, 具体形式可以表示为 [20] 启发, 本文使用参数矩阵的 21  范数作为 稀疏正则, 同时考虑卷积层输出通道对应当前层和 下一层的参数 [21] . 于是, 本文采用的动态稀疏正则 [22] 可以表示为…”