DoubleQExt: Hardware and Memory Efficient CNN Through Two Levels of Quantization

Abstract

To fulfil the tight area and memory constraints in IoT applications, the design of efficient Convolutional Neural Network (CNN) hardware becomes crucial. Quantization of CNN is one of the promising approach that allows the compression of large CNN into a much smaller one, which is very suitable for IoT applications. Among various proposed quantization schemes, Power-of-two (PoT) quantization enables efficient hardware implementation and small memory consumption for CNN accelerators, but requires retraining of CNN to retain its accuracy. This paper proposes a two-level post-training static quantization technique (DoubleQ) that combines the 8-bit and PoT weight quantization. The CNN weight is first quantized to 8-bit (level one), then further quantized to PoT (level two). This allows multiplication to be carried out using shifters, by expressing the weights in their PoT exponent form. DoubleQ also reduces the memory storage requirement for CNN, as only the exponent of the weights is needed for storage. However, DoubleQ trades the accuracy of the network for reduced memory storage. To recover the accuracy, a selection process (DoubleQExt) was proposed to strategically select some of the less informative layers in the network to be quantized with PoT at the second level. On ResNet-20, the proposed DoubleQ can reduce the memory consumption by 37.50% with 7.28% accuracy degradation compared to 8-bit quantization. By applying DoubleQExt, the accuracy is only degraded by 1.19% compared to 8-bit version while achieving a memory reduction of 23.05%. This result is also 1% more accurate than the state-of-the-art work (SegLog). The proposed DoubleQExt also allows flexible configuration to trade off the memory consumption with better accuracy, which is not found in the other state-of-the-art works. With the proposed two-level weight quantization, one can achieve a more efficient hardware architecture for CNN with minimal impact to the accuracy, which is crucial for IoT applications.

FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs With Dynamic Fixed-Point Representation

“…DoubleQExt [88] quantizes weights and activations to 8-bit integers using layer-wise FP32 scalar and offset parameters. Thereafter, it quantizes the integer weights again to represent them in power-of-2 form using 5 bits, thus, reducing computational and memory cost.…”

“…As a summary of the literature, improving the accuracy of quantized DNNs comes at the expense of floating-point computational cost in [30], [32], [34], [35], [38], [42], [45], [56]- [58], [61], [63]- [67], [69], [74], [76], [78]- [80], [82]- [84], [86]- [88]. Specifically, these approaches scale output activations of each layer with FP32 coefficient(s) to recover the dynamic range, and/or perform batch normalization as well as the operations of first and last layers with FP32 datastructures.…”

“…Further, while the latter framework performs the MAC operation using 9.65-bit activations and 8-bit weights, on average, the former is more computationally efficient because it uses 6.29-bit activations and 4.81-bit weights. Compared to DoubleQExt [88], FxP-QNet reduces memory requirements for model parameters by about 1.35MB with only 0.20% and 0.29% extra drop in top-1 and top-5 accuracies, respectively. This is despite the fact that the former framework requires floating-point arithmetic to transform the output of MAC operations, whereas FxP-QNet-quantized ResNet-18 is designed for integer-only deployment.…”

Mohaq: Multi-Objective Hardware-Aware Quantization Of Recurrent Neural Networks

“…Recently, there has been a growing interest in post-training quantization [7,12,12,14,[29][30][31]. Post-training quantization quantizes the pre-trained model parameters without any further retraining epochs after quantization.…”

MOHAQ: Multi-Objective Hardware-Aware Quantization of recurrent neural networks

“…Recently, there has been a growing interest in post-training quantization [7,12,12,14,[29][30][31]. Post-training quantization quantizes the pre-trained model parameters without any further retraining epochs after quantization.…”