Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients

Abstract

Pruning neural networks at initialization would enable us to find sparse models that retain the accuracy of the original network while consuming fewer computational resources for training and inference. However, current methods are insufficient to enable this optimization and lead to a large degradation in model performance. In this paper, we identify a fundamental limitation in the formulation of current methods, namely that their saliency criteria look at a single step at the start of training without taking into account the trainability of the network. While pruning iteratively and gradually has been shown to improve pruning performance, explicit consideration of the training stage that will immediately follow pruning has so far been absent from the computation of the saliency criterion. To overcome the short-sightedness of existing methods, we propose Prospect Pruning (ProsPr), which uses meta-gradients through the first few steps of optimization to determine which weights to prune. ProsPr combines an estimate of the higherorder effects of pruning on the loss and the optimization trajectory to identify the trainable sub-network. Our method achieves state-of-the-art pruning performance on a variety of vision classification tasks, with less data and in a single shot compared to existing pruning-at-initialization methods.

Double Dynamic Sparse Training for GANs

“…SNIP (Lee et al, 2018) is one of the pioneering works which aim to find trainable sub-networks without any training. Some following works (Wang et al, 2020a;Tanaka et al, 2020;de Jorge et al, 2020;Alizadeh et al, 2022) aim to propose different metrics to prune networks at initialization. Among them, Synflow (Tanaka et al, 2020), SPP (Lee et al, 2019), andFORCE (de Jorge et al, 2020) try to address the problem of layer collapse during pruning.…”

A Pruning Method Based on Feature Map Similarity Score

“…Finally, pruning on LeNet-5 and AlexNet can compress the number of parameters to 108 and 17.7 times, respectively, without loss of network performance. N. Lee and M. Alizadeh perform one-shot pruning before the network model is initialized [11,12]. Based on the importance of the weight connection, determine the importance of the weight and remove the weight with low importance.…”

“…Criterion Method [8,9] weights weights magnitude train, prune and fine-tune [10] weights weights magnitude mask learning [11] weights weights magnitude prune and train [12] weights weights magnitude prune and train [14] filters L1 norm train, prune and fine-tune [15] filters filters magnitude group-LASSO regularization [16] filters magnitude of batchnorm parameters train, prune and fine-tune [17] filters output of the next layer train, prune and fine-tune [18] filters geometric median of common information in filters train, prune and fine-tune [19] filters average rank of feature map train, prune and fine-tune [20] filters channel independence train, prune and fine-tune [21,22] filters L p norm train, prune, and fine-tune…”

“…where C F represents the number of filters, K represents the size of the filter, and C I represents the number of the channel. (3) Floating-point operations (FLOPs): Floating-point operations are the number of floating-point operations required in the model, and the calculation method is a formula (12), which can reflect the complexity of the model. Floating-point operations in the model are mainly addition and multiplication operations.…”