Thanks for Nothing: Predicting Zero-Valued Activations with Lightweight Convolutional Neural Networks

Shomron, Gil; Banner, Ron; Shkolnik, Moran; Weiser, Uri

doi:10.1007/978-3-030-58607-2_14

Cited by 20 publications

(11 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The min-max statistics are gathered during a quick preprocessing stage on 2K randomly picked images from the training set. In addition, during preprocessing, we recalibrate the BatchNorm layers' running mean and running variance statistics [25,29,31,32]. In all models, the first convolution layer is left intact, since its input activations, which correspond to the image pixels, do not include many zero values, if any.…”

Section: Methodsmentioning

confidence: 99%

Post-Training Sparsity-Aware Quantization

Shomron¹,

Gabbay²,

Kurzum³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Quantization is a technique used in deep neural networks (DNNs) to increase execution performance and hardware efficiency. Uniform post-training quantization (PTQ) methods are common, since they can be implemented efficiently in hardware and do not require extensive hardware resources or a training set. Mapping FP32 models to INT8 using uniform PTQ yields models with negligible accuracy degradation; however, reducing precision below 8 bits with PTQ is challenging, as accuracy degradation becomes noticeable, due to the increase in quantization noise. In this paper, we propose a sparsity-aware quantization (SPARQ) method, in which the unstructured and dynamic activation sparsity is leveraged in different representation granularities. 4-bit quantization, for example, is employed by dynamically examining the bits of 8-bit values and choosing a window of 4 bits, while first skipping zero-value bits. Moreover, instead of quantizing activation-by-activation to 4 bits, we focus on pairs of 8-bit activations and examine whether one of the two is equal to zero. If one is equal to zero, the second can opportunistically use the other's 4-bit budget; if both do not equal zero, then each is dynamically quantized to 4 bits, as described. SPARQ achieves minor accuracy degradation, 2× speedup over widely used hardware architectures, and a practical hardware implementation. The code is available at https://github.com/gilshm/sparq.Preprint. Under review.

show abstract

Section: Methodsmentioning

confidence: 99%

Post-Training Sparsity-Aware Quantization

Shomron¹,

Gabbay²,

Kurzum³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Another representative work by the same authors [14] follows a different approach to exploit spatial correlation. They design a small CNN, called ZAP, to predict whether individual ReLU inputs in convolutional layers will be positive or negative.…”

Section: Relu Output Prediction Based On Spatial Correlationmentioning

confidence: 99%

Mixture-of-Rookies: Saving DNN Computations by Predicting ReLU Outputs

Pinto¹,

Arnau²,

González³

2022

Preprint

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) are widely used in many applications domains. However, they require a vast amount of computations and memory accesses to deliver outstanding accuracy. In this paper, we propose a scheme to predict whether the output of each ReLu activated neuron will be a zero or a positive number in order to skip the computation of those neurons that will likely output a zero. Our predictor, named Mixture-of-Rookies, combines two inexpensive components. The first one exploits the high linear correlation between binarized (1-bit) and full-precision (8-bit) dot products, whereas the second component clusters together neurons that tend to output zero at the same time. We propose a novel clustering scheme based on analysis of angles, as the sign of the dot product of two vectors depends on the cosine of the angle between them. We implement our hybrid zero output predictor on top of a state-of-the-art DNN accelerator. Experimental results show that our scheme introduces a small area overhead of 5.3% while achieving a speedup of 1.2x and reducing energy consumption by 16.5% on average for a set of diverse DNNs.

show abstract

“…[33] employs reinforcement learning to prune channels, and Refs. [34,35] leverage spatial correlations of CNN OFMs to predict and prune zero-value activations. Further pruning techniques based on weight magnitudes were recently introduced in Refs.…”

Section: Prior Workmentioning

confidence: 99%

Compression of Neural Networks for Specialized Tasks via Value Locality

Gabbay

Shomron

2021

Mathematics

Self Cite

View full text Add to dashboard Cite

Convolutional Neural Networks (CNNs) are broadly used in numerous applications such as computer vision and image classification. Although CNN models deliver state-of-the-art accuracy, they require heavy computational resources that are not always affordable or available on every platform. Limited performance, system cost, and energy consumption, such as in edge devices, argue for the optimization of computations in neural networks. Toward this end, we propose herein the value-locality-based compression (VELCRO) algorithm for neural networks. VELCRO is a method to compress general-purpose neural networks that are deployed for a small subset of focused specialized tasks. Although this study focuses on CNNs, VELCRO can be used to compress any deep neural network. VELCRO relies on the property of value locality, which suggests that activation functions exhibit values in proximity through the inference process when the network is used for specialized tasks. VELCRO consists of two stages: a preprocessing stage that identifies output elements of the activation function with a high degree of value locality, and a compression stage that replaces these elements with their corresponding average arithmetic values. As a result, VELCRO not only saves the computation of the replaced activations but also avoids processing their corresponding output feature map elements. Unlike common neural network compression algorithms, which require computationally intensive training processes, VELCRO introduces significantly fewer computational requirements. An analysis of our experiments indicates that, when CNNs are used for specialized tasks, they introduce a high degree of value locality relative to the general-purpose case. In addition, the experimental results show that without any training process, VELCRO produces a compression-saving ratio in the range 13.5–30.0% with no degradation in accuracy. Finally, the experimental results indicate that, when VELCRO is used with a relatively low compression target, it significantly improves the accuracy by 2–20% for specialized CNN tasks.

show abstract

Thanks for Nothing: Predicting Zero-Valued Activations with Lightweight Convolutional Neural Networks

Cited by 20 publications

References 25 publications

Post-Training Sparsity-Aware Quantization

Post-Training Sparsity-Aware Quantization

Mixture-of-Rookies: Saving DNN Computations by Predicting ReLU Outputs

Compression of Neural Networks for Specialized Tasks via Value Locality

Contact Info

Product

Resources

About