Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection

Abstract

Recent empirical works show that large deep neural networks are often highly redundant and one can find much smaller subnetworks without a significant drop of accuracy. However, most existing methods of network pruning are empirical and heuristic, leaving it open whether good subnetworks provably exist, how to find them efficiently, and if network pruning can be provably better than direct training using gradient descent. We answer these problems positively by proposing a simple greedy selection approach for finding good subnetworks, which starts from an empty network and greedily adds important neurons from the large network. This differs from the existing methods based on backward elimination, which remove redundant neurons from the large network. Theoretically, applying our greedy selection strategy on sufficiently large pre-trained networks guarantees to find small subnetworks with lower loss than networks directly trained with gradient descent. Practically, we improve prior arts of network pruning on learning compact neural architectures on Ima-geNet, including ResNet, MobilenetV2/V3, and ProxylessNet. Our theory and empirical results on MobileNet suggest that we should fine-tune the pruned subnetworks to leverage the information from the large model, instead of re-training from new random initialization as suggested in Liu et al. (2019b).

Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough

“…This is comparable to the asymptotic error obtained by directly training a neural network of size n with gradient descent descent, which is also O(n −1 ) following the mean field analysis of Mei et al (2018); Araújo et al (2019); Sirignano & Spiliopoulos (2019). More recently, Ye et al (2020) proposed the first pruning method that achieves a faster O(n −2 ) error rate and is hence provably better than direct training with gradient descent. See Table 1 for a summary on those works.…”

“…No Over-param Deep Net Baykal et al (2019b); Liebenwein et al (2020) O(n −1 ) Baykal et al (2019a); Mussay et al (2020) O(n −1 ) Ye et al (2020) O(n −2 ) × × This paper O(exp(−cn)) Table 1: Overview on theoretical guaranteed pruning methods. Rate above gives how the error due to pruning decays as the size of the pruned network (n) increases.…”

“…Recently, a line of works on network pruning with theoretical guarantees have emerged, including sensitivity-based methods (Baykal et al, 2019b;Liebenwein et al, 2020), coreset-based methods (Baykal et al, 2019a;Mussay et al, 2020), greedy forward selection (Ye et al, 2020). Both the sensitivity-based and coreset-based methods prune the network by sampling and bound the error caused pruning via concentration inequalities.…”

“…However, the analysis of Ye et al (2020) only applies to two-layer networks and requires the original network to be sufficiently over-parameterized. In this paper, we proposed a new greedy optimization based pruning method, which learns sub-networks of size n with a significantly smaller O(exp(−cn))…”

Effective Sparsification of Neural Networks with Global Sparsity Constraint

“…Network Pruning [10,8,39,21,24,14,41,17,34,30,38] has been extensively studied in recent years to reduce the model size and improve the inference efficiency of deep neural networks. Since it is a widely-recognized property that modern neural networks are always overparameterized, pruning methods are developed to remove unimportant parameters in the fully trained dense networks to alleviate such redundancy.…”