VLSI Implementation of Deep Neural Network Using Integral Stochastic Computing

Abstract

The hardware implementation of deep neural networks (DNNs) has recently received tremendous attention: many applications in fact require high-speed operations that suit a hardware implementation. However, numerous elements and complex interconnections are usually required, leading to a large area occupation and copious power consumption. Stochastic computing has shown promising results for low-power area-efficient hardware implementations, even though existing stochastic algorithms require long streams that cause long latencies. In this paper, we propose an integer form of stochastic computation and introduce some elementary circuits. We then propose an efficient implementation of a DNN based on integral stochastic computing. The proposed architecture has been implemented on a Virtex7 FPGA, resulting in 45% and 62% average reductions in area and latency compared to the best reported architecture in literature. We also synthesize the circuits in a 65 nm CMOS technology and we show that the proposed integral stochastic architecture results in up to 21% reduction in energy consumption compared to the binary radix implementation at the same misclassification rate. Due to fault-tolerant nature of stochastic architectures, we also consider a quasi-synchronous implementation which yields 33% reduction in energy consumption w.r.t. the binary radix implementation without any compromise on performance.

Energy-Efficient Stochastic Computing with Superparamagnetic Tunnel Junctions

“…The origins of stochastic computing lie in the observation that time series data of stochastic spike trains in the brain could be modeled by stochastic jumps from ground to V dd in a logic circuit [58][59][60]. It is no surprise, then, that neural network structures have been implemented successfully and energy efficiently in recent stochastic computing work [61][62][63][64]. Rather than carrying out high level arithmetic and logic operations to "theoretically predict" a neural network's output, stochastic computing implements neuromorphic models of the network in CMOS circuitry.…”

A stochastic-computing based deep learning framework using adiabatic quantum-flux-parametron superconducting technology

“…x [1] x [2] x [3] x [4] x x [3] x [4] x [1] x FC layers. For CONV layers, the number of inputs for inner product computation is less compared with that for FC layers.…”

“…SC is inherently an approximate computing technique [15] [52], and there have been disputes about the suitability of SC for precise computing applications [3]. On the other hand, the DNN inference engine is essentially an approximate computing application.…”

“…This is because the final classification result depends on the relative score/logit values of different classes, instead of absolute values. Recent work [35] [3] have pointed out the suitability of SC for DNN acceleration, and [50] has further proved the equivalence between SC-based DNN and binary neural networks, where the latter originate from deep learning society [11]. All the above discussions suggest the potential to build SC-based DNN acceleration using AQFP technology.…”

Designing reconfigurable large-scale deep learning systems using stochastic computing

“…Notice that for either unipolar or bipolar coding format, the represented number ranges in [0, 1] or [-1, 1]. To represent a number beyond this range, a pre-scaling operation [21] or integer bit-stream based representation [22] can be used to relax this constraint.…”