Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot

Su, Jingtong; Chen, Yihang; Cai, Tianle; Wu, Tianhao; Gao, Ruiqi; Wang, Liwei; Lee, Jason D.

doi:10.48550/arxiv.2009.11094

Cited by 8 publications

(16 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the high-sparsity regime we are interested in, and by significantly redesigning the algorithm of Ramanujan et al (2020), we can find rare gems that can be finetuned to competitive accuracy to IMP with warm-up. Furthermore, our rare gems beat all presented baselines by Frankle et al (2020b); Su et al (2020). In Fig.…”

Section: Introductionmentioning

confidence: 61%

“…Their intuition is that this will lead to masks at initialization that are more amenable to training to high accuracy within a few steps. The authors get promising results, but do not compare extensively against the sanity checks of Frankle et al (2020b) and Su et al (2020), and it is unclear how their scheme (ProsPr) and our presented work compare. In a future version of this manuscript we plan to offer a thorough comparison with ProsPr.…”

Section: Related Workmentioning

confidence: 94%

“…In fact, subsequent work brought the effectiveness of IMP into question. Su et al (2020) showed that even randomly sampled sparse subnetworks at initialization can beat lottery tickets found by IMP as long as the layerwise sparsities are chosen carefully. Gale et al (2019) showed that methods like IMP which train tickets from initialization cannot compete with the accuracy of a model trained with pruning as part of the optimization process.…”

Section: Introductionmentioning

confidence: 99%

“…Extensive follow-up studies propose several heuristics for finding trainable sparse subnetworks at initialization (Lee et al, 2018;Wang et al, 2019;Tanaka et al, 2020). However, subsequent work by Frankle et al (2020b); Su et al (2020) show experimentally that all of these methods fail simple sanity checks. Most methods seem to merely identify good sparsities at each layer, but given those, random sparse subnetworks can be trained to similar or better accuracy.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Rare Gems: Finding Lottery Tickets at Initialization

Sreenivasan¹,

Sohn²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

It has been widely observed that large neural networks can be pruned to a small fraction of their original size, with little loss in accuracy, by typically following a time-consuming "train, prune, re-train" approach. Frankle & Carbin (2018) conjecture that we can avoid this by training lottery tickets, i.e., special sparse subnetworks found at initialization, that can be trained to high accuracy. However, a subsequent line of work presents concrete evidence that current algorithms for finding trainable networks at initialization, fail simple baseline comparisons, e.g., against training random sparse subnetworks. Finding lottery tickets that train to better accuracy compared to simple baselines remains an open problem. In this work, we partially resolve this open problem by discovering rare gems: subnetworks at initialization that attain considerable accuracy, even before training. Refining these rare gems-by means of finetuning-beats current baselines and leads to accuracy competitive or better than magnitude pruning methods.

show abstract

Section: Introductionmentioning

confidence: 61%

Section: Related Workmentioning

confidence: 94%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Rare Gems: Finding Lottery Tickets at Initialization

Sreenivasan¹,

Sohn²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To close with a few pointers to the literature, as Lemma 3.2 is essentially a pruning bound, it is potentially of independent interest; see for instance the literature on lottery tickets and pruning (Frankle and Carbin, 2019;Frankle et al, 2020;Su et al, 2020). Secondly, there is already one generalization bound in the literature which exhibits spectral norms, due to Suzuki et al (2019); unfortunately, it also has an explicit dependence on network width.…”

Section: Direct Uniform Convergence Approach In Theorem 14mentioning

confidence: 99%

Generalization bounds via distillation

Hsu¹,

Ji²,

Telgarsky³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper theoretically investigates the following empirical phenomenon: given a highcomplexity network with poor generalization bounds, one can distill it into a network with nearly identical predictions but low complexity and vastly smaller generalization bounds. The main contribution is an analysis showing that the original network inherits this good generalization bound from its distillation, assuming the use of well-behaved data augmentation. This bound is presented both in an abstract and in a concrete form, the latter complemented by a reduction technique to handle modern computation graphs featuring convolutional layers, fully-connected layers, and skip connections, to name a few. To round out the story, a (looser) classical uniform convergence analysis of compression is also presented, as well as a variety of experiments on cifar10 and mnist demonstrating similar generalization performance between the original network and its distillation.

show abstract

P-PseudoLabel: Enhanced Pseudo-Labeling Framework With Network Pruning in Semi-Supervised Learning

et al. 2022

View full text Add to dashboard Cite

Semi-supervised learning (SSL) methods for classification tasks exhibit a significant performance gain because they combine regularization and pseudo-labeling methods. General pseudolabeling methods only depend on the model's prediction when assigning pseudo-labels, but this approach often leads to the generation of incorrect pseudo-labels, due to the network being biased toward easy classes or to the presence of confusing samples in the training set, which further decreases model performance. To address this issue, we propose a novel pseudo-labeling framework that dramatically reduces the ambiguity of pseudo-labels for confusing samples in SSL. We operate our method, called Pruning for Pseudo-Label (P-PseudoLabel), using the Easy-to-Forget (ETF) Sample Finder, which compares the outputs of the model and the pruned model to identify confusing samples. Next, we perform negative learning using the confusing samples to decrease the risk of providing incorrect information and to improve performance. Our method achieves better performance than those of recent state-of-the-art SSL methods on the CIFAR-10, CIFAR-100, and Mini-ImageNet datasets, and is on par with the state-of-the-art methods on SVHN and STL-10.

show abstract

Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot

Cited by 8 publications

References 20 publications

Rare Gems: Finding Lottery Tickets at Initialization

Rare Gems: Finding Lottery Tickets at Initialization

Generalization bounds via distillation

P-PseudoLabel: Enhanced Pseudo-Labeling Framework With Network Pruning in Semi-Supervised Learning

Contact Info

Product

Resources

About