ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression

Abstract

We propose an efficient and unified framework, namely ThiNet, to simultaneously accelerate and compress CNN models in both training and inference stages. We focus on the filter level pruning, i.e., the whole filter would be discarded if it is less important. Our method does not change the original network structure, thus it can be perfectly supported by any off-the-shelf deep learning libraries. We formally establish filter pruning as an optimization problem, and reveal that we need to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods. Experimental results demonstrate the effectiveness of this strategy, which has advanced the state-of-the-art. We also show the performance of ThiNet on ILSVRC-12 benchmark. ThiNet achieves 3.31× FLOPs reduction and 16.63× compression on VGG-16, with only 0.52% top-5 accuracy drop. Similar experiments with ResNet-50 reveal that even for a compact network, ThiNet can also reduce more than half of the parameters and FLOPs, at the cost of roughly 1% top-5 accuracy drop. Moreover, the original VGG-16 model can be further pruned into a very small model with only 5.05MB model size, preserving AlexNet level accuracy but showing much stronger generalization ability. 1 1 MB = 2 20 ≈ 1.048 million bytes, and 1 million is 10 6 .

Co-Evolutionary Compression for Unpaired Image Translation

“…Interestingly, the discarded filters by our method are not only with small norms but may also have big values, which is significantly different from the results of the conventional filter pruning method, i.e. ThiNet [22]. Actually, the weights in filters for extracting color and texture information can be very small.…”

ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks

“…Therefore, in real-world applications, the developer can use ACNet to enhance a variety of models without exhausting parameter tunings, and the end-users can enjoy the performance improvement without slowing down the inference. Better still, since we introduce no custom structures into the deployed model, it can be future compressed via techniques including connection pruning [9,12], channel pruning [5,6,22,24], quantization [2,10,27], feature map compacting [34], etc.…”

Quantisation and Pruning for Neural Network Compression and Regularisation

“…An iterative pruning method which results in sparse networks is introduced in [14]. ThiNet [18] introduces a filter pruning technique which removes entire filters in convolutional neural networks. The authors propose a method using a least-squares approach to find channels corresponding to unimportant filters.…”