Low-bit Quantization Needs Good Distribution

Yu, Haibao; Wen, Tuopu; Cheng, Guangliang; Sun, Jie; Han, Qi; Shi, Jianping

doi:10.1109/cvprw50498.2020.00348

Cited by 9 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on this, Rueckauer et al ( 2017 ) and Lu and Sengupta ( 2020 ) suggest that a particular percentile from the histogram on the CNN activation may improve conversion efficiency. One step further, Yu et al ( 2020 ) suggest that a good distribution with less outliers on CNN activation can be useful for quantisation. Therefore, we call these techniques distribution-aware CNN training techniques, to emphasise that they are mainly focused on optimizing the CNN training through enforcing good distributions on the activations.…”

Section: Conversion Optimized Through Distribution-aware Cnn Trainingmentioning

confidence: 99%

“…To show that our ECC method is complementary to the distribution-aware CNN training techniques, we implement some existing CNN training techniques that can affect activation distribution and show that ECC can also work with them to achieve optimized conversion. Specifically, Yu et al ( 2020 ) notice that the clipped ReLU can enforce small activation values (i.e., close to zero) to become greater, and eventually reshape the distribution from a Gaussian-like distribution to a uniform distribution. We follow this observation to train CNN models with different clipping methods, including ReLU6 (clipped by 6) from Lin et al ( 2019 ) and Jacob et al ( 2018 ), ReLU-CM (clipped by k-mean) from Yu et al ( 2020 ), and ReLU-SC (shift and clipped) from Deng and Gu ( 2021 ).…”

Section: Conversion Optimized Through Distribution-aware Cnn Trainingmentioning

confidence: 99%

“…Specifically, Yu et al ( 2020 ) notice that the clipped ReLU can enforce small activation values (i.e., close to zero) to become greater, and eventually reshape the distribution from a Gaussian-like distribution to a uniform distribution. We follow this observation to train CNN models with different clipping methods, including ReLU6 (clipped by 6) from Lin et al ( 2019 ) and Jacob et al ( 2018 ), ReLU-CM (clipped by k-mean) from Yu et al ( 2020 ), and ReLU-SC (shift and clipped) from Deng and Gu ( 2021 ). Figure 6A shows that all clipping methods can shift the original activation value (as in the top left figure) to values closer to the mean value (i.e., near the peak area).…”

Section: Conversion Optimized Through Distribution-aware Cnn Trainingmentioning

confidence: 99%

“… Comparison of SNNs using different clipping methods, ReLU6 (Jacob et al, 2018 ; Lin et al, 2019 ), ReLU-CM (Yu et al, 2020 ), ReLU-SC * (Deng and Gu, 2021 ), for CIFAR-10 on VGG-16. (A) Normalized activation distribution of the first layer.…”

Section: Conversion Optimized Through Distribution-aware Cnn Trainingmentioning

confidence: 99%

See 3 more Smart Citations

A Little Energy Goes a Long Way: Build an Energy-Efficient, Accurate Spiking Neural Network From Convolutional Neural Network

Huang

2022

Front. Neurosci.

View full text Add to dashboard Cite

This article conforms to a recent trend of developing an energy-efficient Spiking Neural Network (SNN), which takes advantage of the sophisticated training regime of Convolutional Neural Network (CNN) and converts a well-trained CNN to an SNN. We observe that the existing CNN-to-SNN conversion algorithms may keep a certain amount of residual current in the spiking neurons in SNN, and the residual current may cause significant accuracy loss when inference time is short. To deal with this, we propose a unified framework to equalize the output of the convolutional or dense layer in CNN and the accumulated current in SNN, and maximally align the spiking rate of a neuron with its corresponding charge. This framework enables us to design a novel explicit current control (ECC) method for the CNN-to-SNN conversion which considers multiple objectives at the same time during the conversion, including accuracy, latency, and energy efficiency. We conduct an extensive set of experiments on different neural network architectures, e.g., VGG, ResNet, and DenseNet, to evaluate the resulting SNNs. The benchmark datasets include not only the image datasets such as CIFAR-10/100 and ImageNet but also the Dynamic Vision Sensor (DVS) image datasets such as DVS-CIFAR-10. The experimental results show the superior performance of our ECC method over the state-of-the-art.

show abstract

Section: Conversion Optimized Through Distribution-aware Cnn Trainingmentioning

confidence: 99%

Section: Conversion Optimized Through Distribution-aware Cnn Trainingmentioning

confidence: 99%

Section: Conversion Optimized Through Distribution-aware Cnn Trainingmentioning

confidence: 99%

Section: Conversion Optimized Through Distribution-aware Cnn Trainingmentioning

confidence: 99%

See 2 more Smart Citations

A Little Energy Goes a Long Way: Build an Energy-Efficient, Accurate Spiking Neural Network From Convolutional Neural Network

Huang

2022

Front. Neurosci.

View full text Add to dashboard Cite

show abstract

Search What You Want: Barrier Panelty NAS for Mixed Precision Quantization

Han

et al. 2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Emergent hardwares can support mixed precision CNN models inference that assign different bitwidths for different layers. Learning to find an optimal mixed precision model that can preserve accuracy and satisfy the specific constraints on model size and computation is extremely challenge due to the difficult in training a mixed precision model and the huge space of all possible bit quantizations. In this paper, we propose a novel soft Barrier Penalty based NAS (BP-NAS) for mixed precision quantization, which ensures all the searched models are inside the valid domain defined by the complexity constraint, thus could return an optimal model under the given constraint by conducting search only one time. The proposed soft Barrier Penalty is differentiable and can impose very large losses to those models outside the valid domain while almost no punishment for models inside the valid domain, thus constraining the search only in the feasible domain. In addition, a differentiable Prob-1 regularizer is proposed to ensure learning with NAS is reasonable. A distribution reshaping training strategy is also used to make training more stable. BP-NAS sets new state of the arts on both classification (Cifar-10, ImageNet) and detection (COCO), surpassing all the efficient mixed precision methods designed manually and automatically. Particularly, BP-NAS achieves higher mAP (up to 2.7% mAP improvement) together with lower bit computation cost compared with the existing best mixed precision model on COCO detection.

show abstract