A 42pJ/decision 3.12TOPS/W robust in-memory machine learning classifier with on-chip training

Gonugondla, Sujan K.; Kang, Mingu; Shanbhag, Naresh R.

doi:10.1109/isscc.2018.8310398

Cited by 136 publications

(58 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In another example, the TrueNorth platform (Cassidy et al, 2013 ) is built upon multiple 256 × 256 1-bit synaptic weight crossbars, although it includes extra circuitry to allow assigning up to four possible 8-bit values to the synapses (with some restrictions). In the world of non-spiking Deep Neural Networks (DNN), where there is now a strong quest for providing dedicated efficient hardware (Chen et al, 2016 ; Sim et al, 2016 ; Bong et al, 2017 ; Whatmough et al, 2017 ; Biswas and Chandrakasan, 2018 ; Gonugondla et al, 2018 ; Khwa et al, 2018 ), some theorists are studying ways to reduce bit precision of the weights down to 1-bit (Courbariaux et al, 2015 ; Rastegari et al, 2016 ) to help simplifying hardware. Here we focus on spiking neural network (SNN) hardware capable of on-line unsupervised learning through Spike-Time-Dependent-Plasticity (STDP).…”

Section: Introductionmentioning

confidence: 99%

On Practical Issues for Stochastic STDP Hardware With 1-bit Synaptic Weights

et al. 2018

View full text Add to dashboard Cite

In computational neuroscience, synaptic plasticity learning rules are typically studied using the full 64-bit floating point precision computers provide. However, for dedicated hardware implementations, the precision used not only penalizes directly the required memory resources, but also the computing, communication, and energy resources. When it comes to hardware engineering, a key question is always to find the minimum number of necessary bits to keep the neurocomputational system working satisfactorily. Here we present some techniques and results obtained when limiting synaptic weights to 1-bit precision, applied to a Spike-Timing-Dependent-Plasticity (STDP) learning rule in Spiking Neural Networks (SNN). We first illustrate the 1-bit synapses STDP operation by replicating a classical biological experiment on visual orientation tuning, using a simple four neuron setup. After this, we apply 1-bit STDP learning to the hidden feature extraction layer of a 2-layer system, where for the second (and output) layer we use already reported SNN classifiers. The systems are tested on two spiking datasets: a Dynamic Vision Sensor (DVS) recorded poker card symbols dataset and a Poisson-distributed spike representation MNIST dataset version. Tests are performed using the in-house MegaSim event-driven behavioral simulator and by implementing the systems on FPGA (Field Programmable Gate Array) hardware.

show abstract

Section: Introductionmentioning

confidence: 99%

On Practical Issues for Stochastic STDP Hardware With 1-bit Synaptic Weights

et al. 2018

View full text Add to dashboard Cite

show abstract

“…A method called WAGE has been developed to train DNNs with low bitwidth integers at all stages, including gradients and backpropagated errors (Wu et al, 2018 ). Very recently, training methods were also ported onto specific hardware systems: Gonugondla et al ( 2018 ) present a deep in-memory architecture with on-chip training, which is primarily useful to compensate for PVT variations of the analog circuits.…”

Section: Discussionmentioning

confidence: 99%

Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype

et al. 2018

View full text Add to dashboard Cite

The memory requirement of deep learning algorithms is considered incompatible with the memory restriction of energy-efficient hardware. A low memory footprint can be achieved by pruning obsolete connections or reducing the precision of connection strengths after the network has been trained. Yet, these techniques are not applicable to the case when neural networks have to be trained directly on hardware due to the hard memory constraints. Deep Rewiring (DEEP R) is a training algorithm which continuously rewires the network while preserving very sparse connectivity all along the training procedure. We apply DEEP R to a deep neural network implementation on a prototype chip of the 2nd generation SpiNNaker system. The local memory of a single core on this chip is limited to 64 KB and a deep network architecture is trained entirely within this constraint without the use of external memory. Throughout training, the proportion of active connections is limited to 1.3%. On the handwritten digits dataset MNIST, this extremely sparse network achieves 96.6% classification accuracy at convergence. Utilizing the multi-processor feature of the SpiNNaker system, we found very good scaling in terms of computation time, per-core memory consumption, and energy constraints. When compared to a X86 CPU implementation, neural network training on the SpiNNaker 2 prototype improves power and energy consumption by two orders of magnitude.

show abstract

“…6) we can observe that it has a significant spread from its mean value (σ ≈ 30%µ). Now, when I cell is used to modulate the analog voltage (V a ) on the bit-line [13]- [15], [17], there is a wide variation in the V a value and it cannot be controlled very well. This compromises the computation accuracy and extra algorithmic techniques might be required to compensate for that.…”

Section: Overall Architecturementioning

confidence: 99%

“…However, this would lead to an increase in the number of computations and the energy required. [15] proposed an on-chip training to compensate for chip-to-chip variations. However, this would incur energy and timing penalty required to re-train the network corresponding to every single chip.…”

Section: Overall Architecturementioning

confidence: 99%

CONV-SRAM: An Energy-Efficient SRAM With In-Memory Dot-Product Computation for Low-Power Convolutional Neural Networks

Biswas

Chandrakasan

2019

IEEE J. Solid-State Circuits

311

142

View full text Add to dashboard Cite

This work presents an energy-efficient SRAM with embedded dot-product computation capability, for binary-weight convolutional neural networks. A 10T bit-cell based SRAM array is used to store the 1-b filter weights. The array implements dotproduct as a weighted average of the bit-line voltages, which are proportional to the digital input values. Local integrating ADCs compute the digital convolution outputs, corresponding to each filter. We have successfully demonstrated functionality (> 98% accuracy) with the 10,000 test images in the MNIST handwritten digit recognition dataset, using 6-b inputs/outputs. Compared to conventional full-digital implementations using small bit-widths, we achieve similar or better energy-efficiency, by reducing data transfer, due to the highly parallel in-memory analog computations.

show abstract

A 42pJ/decision 3.12TOPS/W robust in-memory machine learning classifier with on-chip training

Cited by 136 publications

References 3 publications

On Practical Issues for Stochastic STDP Hardware With 1-bit Synaptic Weights

On Practical Issues for Stochastic STDP Hardware With 1-bit Synaptic Weights

Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype

CONV-SRAM: An Energy-Efficient SRAM With In-Memory Dot-Product Computation for Low-Power Convolutional Neural Networks

Contact Info

Product

Resources

About