Training Hardware for Binarized Convolutional Neural Network Based on CMOS Invertible Logic

Abstract

In this paper, we implement fast and power-efficient training hardware for convolutional neural networks (CNNs) based on CMOS invertible logic. The backpropagation algorithm is generally hard to implement in hardware because it requires high-precision floating-point arithmetic. Even though parameters of CNNs can be represented by fixed points or even binary during inference, it is still represented by floating points during training. Our hardware uses low-precision data representation for both inference and training. For hardware implementation, we exploit CMOS invertible logic for training. The use of invertible logic enables logic circuits to compute probabilistic bidirectional operation (forward and backward modes) and can be implemented by stochastic computing. The proposed hardware obtains parameters of neural networks such as weights directly from given data (an input feature map and a true label) without backpropagation. For performance evaluation, the proposed hardware is implemented on an FPGA and trains a binarized 2-layer convolutional neural network model using a modified MNIST dataset. This implementation shows an energy efficiency improvement of approximately 134x compared to that of a CPU implementation that executes the training of the same model as that used in the proposed hardware. Training on the proposed hardware is approximately 40x faster than training on the CPU using the backpropagation algorithm while maintaining almost the same cognition accuracy. INDEX TERMS CMOS invertible logic, Neural network, Stochastic computing.

Sparse Random Signals for Fast Convergence on Invertible Logic

“…In conventional works, dense random signals (r(t)) are used, such as uniform random signals of [-1:+1] [1], [5] or binary random signals of [-1,+1] [3], [4], [6] shown in Fig. 2 (b) and (c).…”

“…The bidirectional computing capability is realized by reducing the network energy to the global minimum energy with noise induced by random signals (e.g., a multiplier can be used as a factorizer in the backward mode). Due to the unique feature, several challenging problems can be quickly solved, such as integer factorization (e.g., cryptography problems [1]) and machine learning (e.g., training neural networks [3], [4]).…”

Hardware Acceleration of Large-Scale CMOS Invertible Logic Based on Sparse Hamiltonian Matrices

“…The spin-gate circuit has been presented [9] for a fullyparallel architecture with a fixed Hamiltonian. Each spin contains a different number of inputs from other spins, where the Hamiltonian coefficients are hardwired for a dedicated application [5]- [7]. The fully-parallel manner restricts the invertible-logic hardware to small-scale Hamiltonians due to area limitations on application specific integrated circuit (ASIC) or field-programmable gate array (FPGA).…”

“…By reducing the network energy using random signals, the bidirectional operation can be realized probabilistically. The unique feature of the bidirectional computing can be applied for solving several critical issues, such as integer factorization (e.g., invertible multiplier operates as factorization at a backward mode) and training neural networks [5]- [7].…”

High Convergence Rates of CMOS Invertible Logic Circuits Based on Many-Body Hamiltonians

“…cryptography problems [2]) and machine learning (e.g. training neural networks [3], [4]). The Hamiltonian is constructed by a network of spins (probabilistic nodes) with interactions among them.…”