Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

Abstract

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources. To address this limitation, we introduce "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35× to 49× without affecting their accuracy. Our method first prunes the network by learning only the important connections. Next, we quantize the weights to enforce weight sharing, finally, we apply Huffman coding. After the first two steps we retrain the network to fine tune the remaining connections and the quantized centroids. Pruning, reduces the number of connections by 9× to 13×; Quantization then reduces the number of bits that represent each connection from 32 to 5. On the ImageNet dataset, our method reduced the storage required by AlexNet by 35×, from 240MB to 6.9MB, without loss of accuracy. Our method reduced the size of VGG-16 by 49× from 552MB to 11.3MB, again with no loss of accuracy. This allows fitting the model into on-chip SRAM cache rather than off-chip DRAM memory. Our compression method also facilitates the use of complex neural networks in mobile applications where application size and download bandwidth are constrained. Benchmarked on CPU, GPU and mobile GPU, compressed network has 3× to 4× layerwise speedup and 3× to 7× better energy efficiency.

Enabling Deep Learning on Edge Devices through Filter Pruning and Knowledge Transfer

“…But on one hand, existing model compression methods focused only on creating compressed models for efficient inference without considering how to compression methods affect the training process (Han et al, 2015;Chen et al, 2015;Kadetotad et al, 2016;Li et al, 2016;Polino et al, 2018), and how to reduce the accuracy loss caused by compression. On the other hand, existing knowledge transfer methods have the following limitations: 1) they still require large student models that are not fit for resources constrained devices (Romero et al, 2014;Li et al, 2019;Yim et al, 2017); 2) they only enable to student model to classify the categories that the models are trained with.…”

“…Without loss of generality, we consider image classification tasks and use ResNet, as an example to discuss our proposed on-device learning solution. Image classification is important for many edge applications, and is also the target task of the related model compression and knowledge distillation works (Hinton et al, 2015;Han et al, 2015;Chen et al, 2015;Polino et al, 2018;Srinivas & Babu, 2015). ResNet is a modern architecture with streamlined convolutional layers.…”

“…Model compression techniques can be broadly classified into three categories, weight sharing, quantization, and pruning techniques. Weight sharing reduces the occu-pied memory by using the same set of weights to represent more than one transformations (Han et al, 2015;Chen et al, 2015). Quantization reduces the size of the model by shrinking the number of bits needed for storing the weights (Han et al, 2015;Kadetotad et al, 2016).…”

“…Weight sharing reduces the occu-pied memory by using the same set of weights to represent more than one transformations (Han et al, 2015;Chen et al, 2015). Quantization reduces the size of the model by shrinking the number of bits needed for storing the weights (Han et al, 2015;Kadetotad et al, 2016). Pruning removes redundant weights or neurons while minimizing accuracy loss.…”

“…To deploy DNNs on resource-constrained devices, there are two general approaches. The first approach aims to compress already-trained models, using techniques such as weights sharing (Chen et al, 2015), quantization (Han et al, 2015;Kadetotad et al, 2016), and pruning (Han et al, 2015;LeCun et al, 1990;Srinivas & Babu, 2015). However, a compressed model generated by these approaches is useful only for inference; it cannot be retrained to capture user-or device-specific requirements or new data available at runtime.…”