Pruning Blocks for CNN Compression and Acceleration via Online Ensemble Distillation

Abstract

In this paper, we propose an online ensemble distillation (OED) method to automatically prune blocks/layers of a target network by transferring the knowledge from a strong teacher in an endto-end manner. To accomplish this, we first introduce a soft mask to scale the output of each block in the target network and enforce the sparsity of the mask by sparsity regularization. Then, a strong teacher network is constructed online by replicating the same target networks and ensembling the discriminative features from each target as its new features. Cooperative learning between multiple target networks and the teacher network is further conducted in a closed-loop form, which improves their performance. To solve the optimization problem in an end-to-end manner, we employ the fast iterative shrinkage-thresholding algorithm to fast and reliably remove the redundant blocks, in which the corresponding soft masks are equal to zero. Compared to other structured pruning methods with iterative fine-tuning, the proposed OED is trained more efficiently in one training cycle. Extensive experiments demonstrate the effectiveness of OED, which can not only simultaneously compress and accelerate a variety of CNN architectures but also enhance the robustness of the pruned networks.INDEX TERMS Fast iterative shrinkage-thresholding algorithm, model compression and acceleration, network pruning, online ensemble distillation.

Progressive Kernel Pruning Based on the Information Mapping Sparse Index for CNN Compression

“…In ResNet-56, the CR and AR of the PKP method are 2.51× and 2.54×, respectively. The classification accuracy is 93.51%, which is a decreases of 0.09% from that of the baseline and is the highest compared to those of the methods in [11], [13], [14], [18], [21]- [23], [41]. The CR of the method in [41] reaches 2.93×, but the classification accuracy is 91.58%, which is a decrease of 2.39%.…”

“…The CR of the method in [41] reaches 2.93×, but the classification accuracy is 91.58%, which is a decrease of 2.39%. The AR of the method in [23] reaches 3.09×, and the classification accuracy is 92.29%, which is a decreases of 1.68%. For ResNet-110, the proposed PKP has the highest CR and AR, reaching 2.32× and 2.18×, respectively, and the classification accuracy decreases by 0.45% from that of the baseline, reaching 93.82%; this is also the highest when compared to the classification accuracies of the methods in [13], [14], [18], [23], [41].…”

“…Zhu et al [37] added decorrelation regularization to SSL [17], further reducing the correlation between filters. Wang et al [23] proposed online ensemble distillation for pruning blocks/layers. Liu et al [38] studied acceleration and pruning from the viewpoint of data differences and proposed instance feature sparse regularization, making the feature maps sparse.…”

“…The sparse models obtained from this pruning method are all irregular convolution kernels, which require special software/hardware for accelerated inference [9]. Structured pruning, including convolution kernel pruning [10]- [12], filter/channel pruning [13]- [22], and block pruning [23], removes structural weights; a pruned model that can achieve accelerated inference does not require specific software/hardware and is thus easy to deploy. In this paper, we mainly discuss convolution kernel pruning.…”

Two-Stage Model Compression and Acceleration: Optimal Student Network for Better Performance

“…Date-Free learning(DFL) [34] combines generative adversarial networks and knowledge distillation to construct a datafree knowledge distillation structure, which can train highperformance student networks without data. Wang et al [47] proposed a one-shot automatic pruning method based on the online ensemble distillation, which pruning removes the redundant structures of CNNs at once in a global way to obtain compact ones without any iterative pruning and retraining.…”