The Difficulty of Training Sparse Neural Networks

Evci, Utku; Pedregosa, Fabián; Gomez, Aidan N.; Elsen, Erich

doi:10.48550/arxiv.1906.10732

Cited by 24 publications

(29 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Starting from scratch, those methods learn to optimize the model weights together with sparse connectivity simultaneously. [23,24] first introduced the Sparse Evolutionary Training (SET) technique [23], reaching superior performance compared to training with fixed sparse connectivity [72,27]. [28][29][30] leverages "weight reallocation" to improve performance of obtained sparse subnetworks.…”

Section: Related Workmentioning

confidence: 99%

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Chen¹,

Gan²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Vision transformers (ViTs) have recently received explosive popularity, but their enormous model sizes and training costs remain daunting. Conventional posttraining pruning often incurs higher training budgets. In contrast, this paper aims to trim down both the training memory overhead and the inference complexity, without sacrificing the achievable accuracy. We launch and report the first-ofits-kind comprehensive exploration, on taking a unified approach of integrating sparsity in ViTs "from end to end". Specifically, instead of training full ViTs, we dynamically extract and train sparse subnetworks, while sticking to a fixed small parameter budget. Our approach jointly optimizes model parameters and explores connectivity throughout training, ending up with one sparse network as the final output. The approach is seamlessly extended from unstructured to structured sparsity, the latter by considering to guide the prune-and-grow of self-attention heads inside ViTs. For additional efficiency gains, we further co-explore data and architecture sparsity, by plugging in a novel learnable token selector to adaptively determine the currently most vital patches. Extensive results on ImageNet with diverse ViT backbones validate the effectiveness of our proposals which obtain significantly reduced computational cost and almost unimpaired generalization. Perhaps most surprisingly, we find that the proposed sparse (co-)training can even improve the ViT accuracy rather than compromising it, making sparsity a tantalizing "free lunch". For example, our sparsified DeiT-Small at (5%, 50%) sparsity for (data, architecture), improves 0.28% top-1 accuracy, and meanwhile enjoys 49.32% FLOPs and 4.40% running time savings. Our codes are available at https://github.com/VITA-Group/SViTE.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Chen¹,

Gan²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, this approach comes with severe drawbacks: 1) It does not allow to design training phase and inference phase independently from each other. This restricts interoperability, hinders efficient training, and compromises performance [7]. 2) Sparsity alone does not necessarily reduce computational cost, as it may require higher accuracy, i.e.…”

Section: Motivationmentioning

confidence: 99%

Linear Computation Coding

Müller¹,

Gäde²,

Bereyhi³

2021

Preprint

View full text Add to dashboard Cite

We introduce the new concept of computation coding. Similar to how rate-distortion theory is concerned with the lossy compression of data, computation coding deals with the lossy computation of functions.Particularizing to linear functions, we present an algorithm to reduce the computational cost of multiplying an arbitrary given matrix with an unknown column vector. The algorithm decomposes the given matrix into the product of codebook and wiring matrices whose entries are either zero or signed integer powers of two.For a typical implementation of deep neural networks, the proposed algorithm reduces the number of required addition units several times. To achieve the accuracy of 16-bit signed integer arithmetic for 4k-vectors, no multipliers and only 1.5 adders per matrix entry are needed.

show abstract

“…There are also works study the vanishing gradient problem (VGP) that appears frequently in deep neural network training. Alford et al [2019], Evci et al [2019] point out that the zero values in full-precision weights lead to VGP when training highly pruned neural networks, which eventually jeopardizes the accuracy of the model. Dong et al [2017], Zhou et al [2017] suggest to maintain a portion of the full-precision weights during quantized network training for improving the performance of the model.…”

Section: Related Workmentioning

confidence: 99%

$S^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks

Li¹,

Liu²,

Yu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Shift neural networks reduce computation complexity by removing expensive multiplication operations and quantizing continuous weights into low-bit discrete values, which are fast and energy efficient compared to conventional neural networks. However, existing shift networks are sensitive to the weight initialization, and also yield a degraded performance caused by vanishing gradient and weight sign freezing problem. To address these issues, we propose S 3 low-bit re-parameterization, a novel technique for training low-bit shift networks. Our method decomposes a discrete parameter in a sign-sparse-shift 3-fold manner. In this way, it efficiently learns a low-bit network with a weight dynamics similar to full-precision networks and insensitive to weight initialization. Our proposed training method pushes the boundaries of shift neural networks and shows 3-bit shift networks out-performs their full-precision counterparts in terms of top-1 accuracy on ImageNet.Preprint. Under review.

show abstract

The Difficulty of Training Sparse Neural Networks

Cited by 24 publications

References 5 publications

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Chasing Sparsity in Vision Transformers: An End-to-End Exploration

Linear Computation Coding

$S^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks

Contact Info

Product

Resources

About