Pruning via Iterative Ranking of Sensitivity Statistics

Stijn, Verdenius,; Stol, Maarten; Forré, Patrick

doi:10.48550/arxiv.2006.00896

Cited by 7 publications

(13 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3). Indeed, very recent work empirically confirms that iteration improves the performance of SNIP [46].…”

Section: Synflow Random Graspmentioning

confidence: 89%

Pruning neural networks without any data by iteratively conserving synaptic flow

Tanaka¹,

Kunin²,

Yamins³

et al. 2020

Preprint

View full text Add to dashboard Cite

Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have identified, through an expensive sequence of training and pruning cycles, the existence of winning lottery tickets or sparse trainable subnetworks at initialization. This raises a foundational question: can we identify highly sparse trainable subnetworks at initialization, without ever training, or indeed without ever looking at the data? We provide an affirmative answer to this question through theory driven algorithm design. We first mathematically formulate and experimentally verify a conservation law that explains why existing gradientbased pruning algorithms at initialization suffer from layer-collapse, the premature pruning of an entire layer rendering a network untrainable. This theory also elucidates how layer-collapse can be entirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow Pruning (SynFlow). This algorithm can be interpreted as preserving the total flow of synaptic strengths through the network at initialization subject to a sparsity constraint. Notably, this algorithm makes no reference to the training data and consistently outperforms existing state-of-the-art pruning algorithms at initialization over a range of models (VGG and ResNet), datasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to 99.9 percent). Thus our data-agnostic pruning algorithm challenges the existing paradigm that data must be used to quantify which synapses are important.

show abstract

“…3). Indeed, very recent work empirically confirms that iteration improves the performance of SNIP [46].…”

Section: Synflow Random Graspmentioning

confidence: 89%

Pruning neural networks without any data by iteratively conserving synaptic flow

Tanaka¹,

Kunin²,

Yamins³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Therefore, more recent works have focused on scoring weights without the need for training using first order (Lee et al, 2019;Tanaka et al, 2020) and second order (Wang et al, 2020;Lubana & Dick, 2021) information. Note that the pruning process can be applied in one-shot or iteratively (de Jorge et al, 2021;Verdenius et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

Winning the Lottery Ahead of Time: Efficient Early Network Pruning

Rachwan¹,

Zügner²,

Charpentier³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pruning, the task of sparsifying deep neural networks, received increasing attention recently. Although state-of-the-art pruning methods extract highly sparse models, they neglect two main challenges: (1) the process of finding these sparse models is often very expensive; (2) unstructured pruning does not provide benefits in terms of GPU memory, training time, or carbon emissions. We propose Early Compression via Gradient Flow Preservation (EarlyCroP), which efficiently extracts state-of-the-art sparse models before or early in training addressing challenge (1), and can be applied in a structured manner addressing challenge (2). This enables us to train sparse networks on commodity GPUs whose dense versions would be too large, thereby saving costs and reducing hardware requirements. We empirically show that EarlyCroP outperforms a rich set of baselines for many tasks (incl. classification, regression) and domains (incl. computer vision, natural language processing, and reinforcment learning). Early-CroP leads to accuracy comparable to dense training while outperforming pruning baselines.

show abstract

“…On the contrary, if too many parameters are removed at one time, almost all models suffer from accuracy drops. This finding makes a connection to the success of the iterative magnitude pruning [10,53,5,6,63], where usually a pruning process with a small pruning rate (e.g., 0.2) needs to be iteratively repeated for good performance.…”

Section: Introductionmentioning

confidence: 99%

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

Liu¹,

Chen

et al. 2021

Preprint

View full text Add to dashboard Cite

Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning (pruning at initialization). The former method suffers from an extremely large computation cost and the latter category of methods usually struggles with insufficient performance. In comparison, during-training pruning, a class of pruning methods that simultaneously enjoys the training/inference efficiency and the comparable performance, temporarily, has been less explored. To better understand during-training pruning, we quantitatively study the effect of pruning throughout training from the perspective of pruning plasticity (the ability of the pruned networks to recover the original performance). Pruning plasticity can help explain several other empirical observations about neural network pruning in literature. We further find that pruning plasticity can be substantially improved by injecting a brain-inspired mechanism called neuroregeneration, i.e., to regenerate the same number of connections as pruned. Based on the insights from pruning plasticity, we design a novel gradual magnitude pruning (GMP) method, named gradual pruning with zero-cost neuroregeneration (GraNet), and its dynamic sparse training (DST) variant (GraNet-ST). Both of them advance state of the art. Perhaps most impressively, the latter for the first time boosts the sparse-to-sparse training performance over various dense-to-sparse methods by a large margin with ResNet-50 on ImageNet. We will release all codes.Preprint. Under review.

show abstract

Pruning via Iterative Ranking of Sensitivity Statistics

Cited by 7 publications

References 42 publications

Pruning neural networks without any data by iteratively conserving synaptic flow

Pruning neural networks without any data by iteratively conserving synaptic flow

Winning the Lottery Ahead of Time: Efficient Early Network Pruning

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration

Contact Info

Product

Resources

About