Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Abstract

The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components. Similarly to their biological counterparts, sparse networks generalize just as well, if not better than, the original dense networks. Sparsity can reduce the memory footprint of regular networks to fit mobile devices, as well as shorten training time for ever growing networks. In this paper, we survey prior work on sparsity in deep learning and provide an extensive tutorial of sparsification for both inference and training. We describe approaches to remove and add elements of neural networks, different training strategies to achieve model sparsity, and mechanisms to exploit sparsity in practice. Our work distills ideas from more than 300 research papers and provides guidance to practitioners who wish to utilize sparsity today, as well as to researchers whose goal is to push the frontier forward. We include the necessary background on mathematical methods in sparsification, describe phenomena such as early structure adaptation, the intricate relations between sparsity and the training process, and show techniques for achieving acceleration on real hardware. We also define a metric of pruned parameter efficiency that could serve as a baseline for comparison of different sparse networks. We close by speculating on how sparsity can improve future workloads and outline major open problems in the field.. The supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience -Albert Einstein, 1933 INTRODUCTIONDeep learning shows unparalleled promise for solving very complex real-world problems in areas such as computer vision, natural language processing, knowledge representation, recommendation systems, drug discovery, and many more. With this development, the field of machine learning is moving from traditional feature engineering to neural architecture engineering. However, still little is known about how to pick the right architecture to solve a specific task. Several methods such as translational equivariance in convolutional layers, recurrence, structured weight sharing, pooling, or locality are used to introduce strong inductive biases in the model design. Yet, the exact model size and capacity required for a task remain unknown and a common strategy is to train overparameterized models and compress them into smaller representations. Biological brains, especially the human brain, are hierarchical, sparse, and recurrent structures [Friston 2008] and one can draw some similarities with the inductive biases in today's artificial neural networks. Sparsity plays an important role in scaling biological brains-the more

Near-Optimal Sparse Allreduce for Distributed Deep Learning

“…We assume all sparse approaches use the coordinate (COO) format to store the sparse gradient, which consumes 2𝑘 storage, i.e., 𝑘 values plus 𝑘 indexes. There are other sparse formats (see [22] for an overview), but format selection for a given sparsity is not the topic of this work. To model the communication overhead, we assume bidirectional and direct point-to-point communication between the compute nodes, and use the classic latency-bandwidth cost model.…”

“…Only the nonzero values of the distributed gradients are accumulated across all processes. See [22] for an overview of gradient and other sparsification approaches in deep learning.…”

How Well Do Sparse Imagenet Models Transfer?

“…Recently, there has been significant research interest in pruning techniques, and hundreds of different sparsification approaches have been proposed; please see the recent surveys of [15] and [25] for a comprehensive exposition. We categorize existing pruning methods as follows.…”

“…The increasing computational and storage costs of deep learning models have led to significant academic and industrial interest in model compression, which is roughly the task of obtaining smaller-footprint models matching the accuracy of larger baseline models. Model compression is a rapidly-developing area, and several generic approaches have been investigated, among which pruning and quantization are among the most popular [16,25].…”

“…objective is to remove, by setting to zero, as many weights as possible without losing model accuracy. Weight pruning is, arguably, the compression method with the richest history [35] and is currently a very active research topic [25]. Thanks to this trend, a set of fairly consistent accuracy benchmarks has emerged for pruning, along with increasingly efficient computational support [9,18,33,43].…”