Learning Channel-Wise Interactions for Binary Convolutional Neural Networks

Wang, Ziwei; Lu, Jiwen; Tao, Chenxin; Zhou, Jie; Tian, Qi

doi:10.1109/cvpr.2019.00066

Cited by 84 publications

(15 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Top-5 Acc(%) CI-BCNN (Wang et al, 2019) One step 59.9 84.2 Binary MobileNet (Phan et al, 2020b) 60.9 82.6 MoBiNet (Phan et al, 2020a) 54.4 77.5 EL (Hu et al, 2022) 56.4 -MeliusNet29 (Bethge et al, 2020) 65.8 -ReActNet-A + Our filtering optimizer 69.7 88.9…”

Section: Methods Training Strategymentioning

confidence: 99%

Understanding weight-magnitude hyperparameters in training binary networks

Quist¹,

Li²,

Gemert³

2023

Preprint

View full text Add to dashboard Cite

Binary Neural Networks (BNNs) are compact and efficient by using binary weights instead of real-valued weights. Current BNNs use latent real-valued weights during training, where hyper-parameters are inherited from real-valued networks. The interpretation of several of these hyperparameters is based on the magnitude of the real-valued weights. For BNNs, however, the magnitude of binary weights is not meaningful, and thus it is unclear what these hyperparameters actually do. One example is weight-decay, which aims to keep the magnitude of real-valued weights small. Other examples are latent weight initialization, the learning rate, and learning rate decay, which influence the magnitude of the real-valued weights. The magnitude is interpretable for real-valued weights, but loses its meaning for binary weights. In this paper we offer a new interpretation of these magnitude-based hyperparameters based on higher-order gradient filtering during network optimization. Our analysis makes it possible to understand how magnitude-based hyperparameters influence the training of binary networks which allows for new optimization filters specifically designed for binary neural networks that are independent of their real-valued interpretation. Moreover, our improved understanding reduces the number of hyperparameters, which in turn eases the hyperparameter tuning effort which may lead to better hyperparameter values for improved accuracy.

show abstract

Section: Methods Training Strategymentioning

confidence: 99%

Understanding weight-magnitude hyperparameters in training binary networks

Quist¹,

Li²,

Gemert³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The knowledge transfer is learning the class distribution output via the loss function. Therefore, the BNN can be under the supervision of a real-value model to improve the learning capability to gain higher accuracy that is closed to the real-valued model like CI-BCNN [69]. CI-BCNN extracts the channelwise interactions from the prior knowledge to decrease the inconsistency of signs in binary feature maps and keeps the information of input samples during inference.…”

Section: ) Accuracy Optimization Approachesmentioning

confidence: 99%

A Systematic Literature Review on Binary Neural Networks

et al. 2023

View full text Add to dashboard Cite

This paper presents an extensive literature review on Binary Neural Network (BNN). BNN utilizes binary weights and activation function parameters to substitute the full-precision values. In digital implementations, BNN replaces the complex calculations of Convolutional Neural Networks (CNNs) with simple bitwise operations. BNN optimizes large computation and memory storage requirements, which leads to less area and power consumption compared to full-precision models. Although there are many advantages of BNN, the binarization process has a significant impact on the performance and accuracy of the generated models. To reflect the state-of-the-art in BNN and explore how to develop and improve BNNbased models, we conduct a systematic literature review on BNN with data extracted from 239 research studies. Our review discusses various BNN architectures and the optimization approaches developed to improve their performance. There are three main research directions in BNN: accuracy optimization, compression optimization, and acceleration optimization. The accuracy optimization approaches include quantization error reduction, special regularization, gradient error minimization, and network structure. The compression optimization approaches combine fractional BNN and pruning. The acceleration optimization approaches comprise computing in-memory, FPGA-based implementations, and ASIC-based implementations. At the end of our review, we present a comprehensive analysis of BNN applications and their evaluation metrics. Also, we shed some light on the most common BNN challenges and the future research trends of BNN.

show abstract

“…Unlike optimizing the binary process directly at the convolution layer, LAB2 [4] directly considers the binary loss and applies the near-end Newton algorithm to the binary weights. CI-BCNN [5], through learning to strengthen graph models, mining channel-level interaction, and iterating pop count, reduces symbol inconsistency in binary feature graphs and retains the input sample information. LNS [6] proposes to predict binary weight by monitoring noise learning and training binary functions.…”

Section: Related Workmentioning

confidence: 99%

A Lightweight Binarized Convolutional Neural Network Model for Small Memory and Low-Cost Mobile Devices

Sun

Mei

et al. 2023

Mobile Information Systems

View full text Add to dashboard Cite

In recent years, the high cost of implementing deep neural networks due to their large model size and parameter complexity has made it a challenging problem to design lightweight models that reduce application costs. The existing binarized neural networks suffer from both the large memory occupancy and the big number of trainable params they use. We propose a lightweight binarized convolutional neural network (CBCNN) model to address the multiclass classification/identification problem. We use both binary weights and activation. We show experimentally that a model using only 0.59 M trainable params is sufficient to reach about 92.94% accuracy on the GTSRB dataset, and it has similar performances compared to other methods on MNIST and Fashion-MNIST datasets. Accordingly, most arithmetic operations with bitwise operations are simplified, thus both used memory size and memory accesses are reduced by 32 times. Moreover, the color information was removed, which also reduced drastically the training time. All these together allow our architecture to run effectively and in real time on simple CPUs (rather than GPUs). Through the results we obtained, we show that despite simplifications and color information removal, our network achieves similar performances compared to classical CNNs with lower costs in both in training and embedded deployment.

show abstract

Learning Channel-Wise Interactions for Binary Convolutional Neural Networks

Cited by 84 publications

References 24 publications

Understanding weight-magnitude hyperparameters in training binary networks

Understanding weight-magnitude hyperparameters in training binary networks

A Systematic Literature Review on Binary Neural Networks

A Lightweight Binarized Convolutional Neural Network Model for Small Memory and Low-Cost Mobile Devices

Contact Info

Product

Resources

About