Pruning from Scratch

Abstract

Network pruning is an important research field aiming at reducing computational costs of neural networks. Conventional approaches follow a fixed paradigm which first trains a large and redundant network, and then determines which units (e.g., channels) are less important and thus can be removed. In this work, we find that pre-training an over-parameterized model is not necessary for obtaining the target pruned structure. In fact, a fully-trained over-parameterized model will reduce the search space for the pruned structure. We empirically show that more diverse pruned structures can be directly pruned from randomly initialized weights, including potential models with better performance. Therefore, we propose a novel network pruning pipeline which allows pruning from scratch with little training overhead. In the experiments for compressing classification models on CIFAR10 and ImageNet datasets, our approach not only greatly reduces the pre-training burden of traditional pruning methods, but also achieves similar or even higher accuracy under the same computation budgets. Our results facilitate the community to rethink the effectiveness of existing techniques used for network pruning.

Accelerating Federated Learning for IoT in Big Data Analytics With Pruning, Quantization and Selective Updating

“…(3) Unlike [7], only oneshot pruning is adopted by FL-PQSU, as the further pruning in federated training incurs additional overhead, but contributes little to performance improvement [13]. According to our testing experiments, 1 -norm based pruning outperforms blind model shrinking by about 1% accuracy loss for large models like VGG16, which can't be simply neglected [16].…”

“…Unlike these previous approaches, Play and Prune [15] allows to specify the error tolerance limit instead of the pruning ratio for each layer. Wang et al [16] verify that pruning from randomly initialized weights directly can result in more diverse pruned structures with competitive performance. More recent studies [7], [17] proposed to prune the model during training for not only improving the performance of inference but also reducing the costs of training.…”

“…This is the most common case in FL [3]. Fortunately, according to [16], the model pruning can be directly started from randomly initialized weights, and model convergence is guaranteed.…”

Tensor Reordering for CNN Compression

“…The implementation of our proposed compression technique 1 is assessed in terms of accuracy vs. parameter count against state-ofthe-art pruning approaches including [1,2,4,7,8,10,15,21,22,23]. We present strong results with our novel CNN compression applied to ResNet-50 [12] and MobileNet-V2 [13] architectures for image classification task on ImageNet ILSVRC dataset [24] (Sec.…”

KecNet: A Light Neural Network for Arrhythmia Classification Based on Knowledge Reinforcement

“…Currently, there exist two main approaches to build lightweight networks. (1) The first approach uses network pruning techniques [ 9 ] or knowledge distillation [ 10 ] to achieve model compression and inference acceleration by removing redundant structures and parameters. Because the accuracy-focused models contain strategies that help overcome various problems encountered during training, such as overfitting, it is difficult to scale such a model down sufficiently without sacrificing accuracy.…”