Extremely Low-bit Convolution Optimization for Quantized Neural Network on Modern Computer Architectures

Han, Qingchang; Hu, Yongmin; Yu, Fei; Yang, Hailong; Liu, Bing; Hu, Peng; Gong, Ruihao; Wang, Yanfei; Wang, Rui; Luan, Zhongzhi; Qian, Depei

doi:10.1145/3404397.3404407

Cited by 14 publications

(9 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Layer-wise Reconstruction: Assume the Hessian matrix is layer-diagonal and optimize the layer output one-by-one. It does not consider cross-layer dependency and resemble existing methods (Nagel et al, 2020;Hubara et al, 2020;Wang et al, 2020). 2.…”

Section: Block Reconstructionmentioning

confidence: 99%

See 1 more Smart Citation

BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

Li¹,

Gong²,

Tan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We study the challenging task of neural network quantization without end-toend retraining, called Post-training Quantization (PTQ). PTQ usually requires a small subset of training data but produces less powerful quantized models than Quantization-Aware Training (QAT). In this work, we propose a novel PTQ framework, dubbed BRECQ, which pushes the limits of bitwidth in PTQ down to INT2 for the first time. BRECQ leverages the basic building blocks in neural networks and reconstructs them one-by-one. In a comprehensive theoretical study of the second-order error, we show that BRECQ achieves a good balance between crosslayer dependency and generalization error. To further employ the power of quantization, the mixed precision technique is incorporated in our framework by approximating the inter-layer and intra-layer sensitivity. Extensive experiments on various handcrafted and searched neural architectures are conducted for both image classification and object detection tasks. And for the first time we prove that, without bells and whistles, PTQ can attain 4-bit ResNet and MobileNetV2 comparable with QAT and enjoy 240× faster production of quantized models. Codes are available at https://github.com/yhhhli/BRECQ.

show abstract

Section: Block Reconstructionmentioning

confidence: 99%

“…For the acquisition of mobile ARM CPU latency, we adopt the redesigned low-bit GEMM implementation in Han et al (2020). Fig.…”

Section: B43 Latency Acquisitionmentioning

confidence: 99%

BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

Li¹,

Gong²,

Tan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…[45] embed linear quantization directly into the Winograd domain to achieve low-precision quantization. [46] further explored the use of the Winograd algorithm to optimize the convolution kernel with 4-6bit precision. [47] applied quantization on feature map slices, and applied particle swarm optimization technology to find the threshold of quantization.…”

Section: Low Precision and Quantizationmentioning

confidence: 99%

Fast Convolution based on Winograd Minimum Filtering: Introduction and Development

Gan¹,

Huang²

2021

Computer Science and Information Technology Trends

View full text Add to dashboard Cite

Convolutional Neural Network (CNN) has been widely used in various fields and played an important role. Convolution operators are the fundamental component of convolutional neural networks, and it is also the most time-consuming part of network training and inference. In recent years, researchers have proposed several fast convolution algorithms including FFT and Winograd. Among them, Winograd convolution significantly reduces the multiplication operations in convolution, and it also takes up less memory space than FFT convolution. Therefore, Winograd convolution has quickly become the first choice for fast convolution implementation within a few years. At present, there is no systematic summary of the convolution algorithm. This article aims to fill this gap and provide detailed references for follow-up researchers. This article summarizes the development of Winograd convolution from the three aspects of algorithm expansion, algorithm optimization, implementation, and application, and finally makes a simple outlook on the possible future directions.

show abstract

“…Then, it is difficult to run large-scale DNNs on mobile devices. In order to use DNNs on mobile devices, a number of methods have been developed [4,8].…”

Section: Introductionmentioning

confidence: 99%

Development of Quantized DNN Library for Exact Hardware Emulation

Kiyama,

Amagasaki,

Iida

2021

Preprint

View full text Add to dashboard Cite

Quantization is used to speed up execution time and save power when runnning Deep neural networks (DNNs) on edge devices like AI chips. To investigate the effect of quantization, we need performing inference after quantizing the weights of DNN with 32-bit floating-point precision by a some bit width, and then quantizing them back to 32-bit floating-point precision. This is because the DNN library can only handle floating-point numbers. However, the accuracy of the emulation does not provide accurate precision. We need accurate precision to detect overflow in MAC operations or to verify the operation on edge devices. We have developed PyParch, a DNN library that executes quantized DNNs (QNNs) with exactly the same behavior as hardware. In this paper, we describe a new proposal and implementation of PyParch. As a result of the evaluation, the accuracy of QNNs with arbitrary bit widths can be estimated for large and complex DNNs such as YOLOv5, and the overflow can be detected. We evaluated the overhead of the emulation time and found that it was 5.6 times slower for QNN and 42 times slower for QNN with overflow detection compared to the normal DNN execution time.

show abstract

Extremely Low-bit Convolution Optimization for Quantized Neural Network on Modern Computer Architectures

Cited by 14 publications

References 12 publications

BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

Fast Convolution based on Winograd Minimum Filtering: Introduction and Development

Development of Quantized DNN Library for Exact Hardware Emulation

Contact Info

Product

Resources

About