Self-Supervised Quantization of Pre-Trained Neural Networks for Multiplierless Acceleration

Vogel, Sebastian; Springer, Jannik; Guntoro, Andre; Ascheid, Gerd

doi:10.23919/date.2019.8714901

Cited by 23 publications

(7 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We will elaborate and validate the proposed method in the following sections. In the network quantization area, Vogel et al [36] presented a non-retraining method for quantizing networks. This may be the closest work to our study.…”

Section: Related Workmentioning

confidence: 99%

S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration

Shen

Liu

Qin

et al. 2021

Preprint

View full text Add to dashboard Cite

Previous studies dominantly target at self-supervised learning on real-valued networks and have achieved many promising results. However, on the more challenging binary neural networks (BNNs), this task has not yet been fully explored in the community. In this paper, we focus on this more difficult scenario: learning networks where both weights and activations are binary, meanwhile, without any human annotated labels. We observe that the commonly used contrastive objective is not satisfying on BNNs for competitive accuracy, since the backbone network contains relatively limited capacity and representation ability. Hence instead of directly applying existing self-supervised methods, which cause a severe decline in performance, we present a novel guided learning paradigm from realvalued to distill binary networks on the final prediction distribution, to minimize the loss and obtain desirable accuracy. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.5∼15% on BNNs. We further reveal that it is difficult for BNNs to recover the similar predictive distributions as real-valued models when training without labels. Thus, how to calibrate them is key to address the degradation in performance. Extensive experiments are conducted on the large-scale Ima-geNet and downstream datasets. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods. Code will be made available.

show abstract

Section: Related Workmentioning

confidence: 99%

S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration

Shen

Liu

Qin

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, most of these techniques require a fine-tuning (retraining) step to reduce the errors induced due to quantization. The authors of [17] have avoided the post-quantization fine-tuning step by computing the quantization step size using an iterative approach. In their proposed technique, the optimal quantization step sizes for features and parameters are computed by iteratively adjusting the step size for each data structure in each layer and recording the generated errors in the layer under consideration.…”

Section: Related Workmentioning

confidence: 99%

“…-bits [17] divides the float32 parameter into two segments i.e., MSBs and LSBs. It then performs the log 2 quantization of the MSBs and LSBs separately and then adds the MSBs-quantized and LSBs-quantized values to achieve the final quantized value.…”

Section: S Leading One Locationmentioning

confidence: 99%

“…These images are classified into 1000 different categories. For our experimentation, we have taken pre-trained AlexNet [28], VGG-16 [17] and ResNet-18 networks [30]. To improve the inference accuracy of power of 2 quantization scheme, we have also utilized the postquantization retraining for it to minimize its quantization induced errors.…”

Section: A Image Classificationmentioning

confidence: 99%

See 1 more Smart Citation

ALigN: A Highly Accurate Adaptive Layerwise Log_2_Lead Quantization of Pre-Trained Neural Networks

et al. 2020

View full text Add to dashboard Cite

Deep Neural Networks are one of the machine learning techniques which are increasingly used in a variety of applications. However, the significantly high memory and computation demands of deep neural networks often limit their deployment on embedded systems. Many recent works have considered this problem by proposing different types of data quantization schemes. However, most of these techniques either require post-quantization retraining of deep neural networks or bear a significant loss in output accuracy. In this paper, we propose a novel and scalable technique with two different modes for the quantization of the parameters of pre-trained neural networks. In the first mode, referred to as log_2_lead, we use a single template for the quantization of all parameters. In the second mode, denoted as ALigN, we analyze the trained parameters of each layer and adaptively adjust the quantization template to achieve even higher accuracy. Our technique significantly maintains the accuracy of the parameters and does not require retraining of the networks. Moreover, it supports quantization to an arbitrary bit-size. For example, compared to the single-precision floating-point numbers-based implementation, our proposed 8bit quantization technique generates only ∼ 0.2% and ∼ 0.1%, loss in the Top-1 and Top-5 accuracies respectively for VGG-16 network using ImageNet dataset. We have observed similar minimal losses in the Top-1 and Top-5 accuracies for AlexNet and Resnet-18 using the proposed quantization scheme for the 8-bit range. Our proposed quantization technique also provides a higher mean intersection over union for semantic segmentation when compared with state-of-the-art quantization techniques. The proposed technique represents parameters in powers of 2, thereby eliminating the need for resource-computationally intensive multiplier units for the hardware accelerators of the neural networks. We also present a design for implementing the multiplication operation using bit-shifts and addition for the proposed quantization technique.

show abstract

“…A quantization method which specifically targets this problem has been introduced in [94]. Here, parameters and activations are quantized by minimizing the effect of the quantization error δ = χ − χ q in the network.…”

Section: Fixed-point Quantizationmentioning

confidence: 99%

Automated design of error-resilient and hardware-efficient deep neural networks

Schorn¹,

Elsken²,

Vogel³

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Applying deep neural networks (DNNs) in mobile and safety-critical systems, such as autonomous vehicles, demands a reliable and efficient execution on hardware. Optimized dedicated hardware accelerators are being developed to achieve this. However, the design of efficient and reliable hardware has become increasingly difficult, due to the increased complexity of modern integrated circuit technology and its sensitivity against hardware faults, such as random bit-flips. It is thus desirable to exploit optimization potential for error resilience and efficiency also at the algorithmic side, e.g. by optimizing the architecture of the DNN. Since there are numerous design choices for the architecture of DNNs, with partially opposing effects on the preferred characteristics (such as small error rates at low latency), multi-objective optimization strategies are necessary. In this paper, we develop an evolutionary optimization technique for the automated design of hardware-optimized DNN architectures. For this purpose, we derive a set of easily computable objective functions, which enable the fast evaluation of DNN architectures with respect to their hardware efficiency and error resilience solely based on the network topology. We observe a strong correlation between predicted error resilience and actual measurements obtained from fault injection simulations. Furthermore, we analyze two dif-

show abstract

Self-Supervised Quantization of Pre-Trained Neural Networks for Multiplierless Acceleration

Cited by 23 publications

References 14 publications

S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration

S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration

ALigN: A Highly Accurate Adaptive Layerwise Log_2_Lead Quantization of Pre-Trained Neural Networks

Automated design of error-resilient and hardware-efficient deep neural networks

Contact Info

Product

Resources

About