Collective Knowledge Workflow for Highly Efficient 8-bit Low Precision Inference of Convolutional Neural Networks with IntelCaffe

Gong, Jianming; Shen, Haihao; Zhang, Guoming; Liu, Xiaoli; Li, Shane; Jin, Ge; Maheshwari, Niharika; Fomenko, Evarist; Segal, Eden

doi:10.1145/3229769

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The influence of parameters quantization on the NN model performance is an active area of research, where the NN parameters have been quantized with 16-bits [20,32], 8-bits [1,9], 4-bits [3] or 2-bits [4]. Moreover, ternary [34] and binary (1-bit) quantization [10,22] have also been taken into account.…”

Section: Introductionmentioning

confidence: 99%

“…The main advantage of USQ is the design simplicity accompanied with relatively good performance when compared to more complex non-uniform quantization. Nevertheless, a detailed design process of the quantizer, taking into account the assumed statistical distribution of NN parameters, is missing in above mentioned papers [1,4,9,10,20,32,34] about quantization of NN parameters. In this paper we design USQ for compression of NN weights assuming Laplacian distribution of weights and bit rates from 9 to 16 bps.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Quantization of Weights of Neural Networks with Negligible Decreasing of Prediction Accuracy

Perić¹,

Denic

Savic³

et al. 2021

ITC

View full text Add to dashboard Cite

Quantization and compression of neural network parameters using the uniform scalar quantization is carried out in this paper. The attractiveness of the uniform scalar quantizer is reflected in a low complexity and relatively good performance, making it the most popular quantization model. We present a design approach for the memoryless Laplacian source with zero-mean and unit variance, which is based on iterative rule and uses the minimal mean-squared error distortion as a performance criterion. In addition, we derive closed-form expressions for SQNR (Signal to Quantization Noise Ratio) in a wide dynamic range of variance of input data. To show effectiveness on real data, the proposed quantizer is used to compress the weights of neural networks using bit rates from 9 to 16 bps (bits/sample) instead of standardly used 32 bps full precision bit rate. The impact of weights compression on the NN (neural network) performance is analyzed, indicating good matching with the theoretical results and showing negligible decreasing of the prediction accuracy of the NN even in the case of high variance-mismatch between the variance of NN weights and the variance used for the design of quantizer, if the value of the bit-rate is properly chosen according to the rule proposed in the paper.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Quantization of Weights of Neural Networks with Negligible Decreasing of Prediction Accuracy

Perić¹,

Denic

Savic³

et al. 2021

ITC

View full text Add to dashboard Cite

show abstract

Design and Analysis of Binary Scalar Quantizer of Laplacian Source with Applications

Perić

Denic

Savic³

et al. 2020

Information

View full text Add to dashboard Cite

A compression method based on non-uniform binary scalar quantization, designed for the memoryless Laplacian source with zero-mean and unit variance, is analyzed in this paper. Two quantizer design approaches are presented that investigate the effect of clipping with the aim of reducing the quantization noise, where the minimal mean-squared error distortion is used to determine the optimal clipping factor. A detailed comparison of both models is provided, and the performance evaluation in a wide dynamic range of input data variances is also performed. The observed binary scalar quantization models are applied in standard signal processing tasks, such as speech and image quantization, but also to quantization of neural network parameters. The motivation behind the binary quantization of neural network weights is the model compression by a factor of 32, which is crucial for implementation in mobile or embedded devices with limited memory and processing power. The experimental results follow well the theoretical models, confirming their applicability in real-world applications.

show abstract

Collective Knowledge Workflow for Highly Efficient 8-bit Low Precision Inference of Convolutional Neural Networks with IntelCaffe

Cited by 2 publications

References 0 publications

Quantization of Weights of Neural Networks with Negligible Decreasing of Prediction Accuracy

Quantization of Weights of Neural Networks with Negligible Decreasing of Prediction Accuracy

Design and Analysis of Binary Scalar Quantizer of Laplacian Source with Applications

Contact Info

Product

Resources

About