Adaptive learning rule for hardware-based deep neural networks using electronic synapse devices

Lim, Suhwan; Bae, Jong-Ho; Eum, Jai-Ho; Lee, Sungtae; Kim, Chul-Heung; Kwon, Dongseok; Park, Byung‐Gook; Lee, Jong-Ho

doi:10.1007/s00521-018-3659-y

Cited by 58 publications

(40 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For high performance in HNNs, various synaptic arrays and neuron circuits suitable for efficient architectures and learning algorithms have been researched [4][5][6][7][8]. Specifically, both excitatory (G + ) and inhibitory (G -) synaptic arrays are important to improve the accuracy of HNNs [9][10][11]. Neuron circuits that use large capacitors (≥ 0.1 pF) and many transistors (≥ 11 MOSFETs) to process simultaneously signals from these two types of synapses have been reported [11][12][13], resulting in increased power consumption and a larger area.…”

Section: Introductionmentioning

confidence: 99%

Low-Power and High-Density Neuron Device for Simultaneous Processing of Excitatory and Inhibitory Signals in Neuromorphic Systems

et al. 2020

Self Cite

View full text Add to dashboard Cite

A positive-feedback (PF) neuron device capable of threshold tuning and simultaneously processing excitatory (G +) and inhibitory (G-) signals is experimentally demonstrated to replace conventional neuron circuits, for the first time. Thanks to the PF operation, the PF neuron device with steep switching characteristics can implement integrate-and-fire (IF) function of neurons with low-energy consumption. The structure of the PF neuron device efficiently merges a gated PNPN diode and a single MOSFET. Integrateand-fire (IF) operation with steep subthreshold swing (SS < 1 mV/dec) is experimentally implemented by carriers accumulated in an n floating body of the PF neuron device. The carriers accumulated in the n floating body are discharged by an inhibitory signal applied to the merged FET. Moreover, the threshold voltage (Vth) of the proposed PF neuron is controlled by using a charge storage layer. The low-energy consuming PF neuron circuit (~ 0.62 pJ/spike) consists of one PF device and only five MOSFETs for the IF and reset operation. In a high-level system simulation, a deep-spiking neural network (D-SNN) based on PF neurons with four hidden layers (1024 neurons in each layer) shows high-accuracy (98.55%) during a MNIST classification task. The PF neuron device provides a viable solution for high-density and low-energy neuromorphic systems.

show abstract

Section: Introductionmentioning

confidence: 99%

Low-Power and High-Density Neuron Device for Simultaneous Processing of Excitatory and Inhibitory Signals in Neuromorphic Systems

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…By using sign of W ij , we can update the conductance of synaptic devices. Weight (W ij ) of synaptic device can be modified by one step (| G − ij |, −| G + ij |) at each iteration according to sign of W ij to reduce the burden of periphery circuit [14].…”

Section: Operation Scheme Of Multi-layer Neural Networkmentioning

confidence: 99%

“…We also investigated the device variation by measuring NAND flash cells and checked the reliability of NAND flash cells by measuring endurance and retention characteristics. Using a matched computer simulation, a 3-layer perceptron network with 40,545 synapses is trained using the weight update method in [14] appropriate for our device and the MNIST data set.…”

Section: Introductionmentioning

confidence: 99%

Operation Scheme of Multi-Layer Neural Networks Using NAND Flash Memory as High-Density Synaptic Devices

Lee

Lim

Choi

et al. 2019

IEEE J. Electron Devices Soc.

Self Cite

View full text Add to dashboard Cite

We propose a designing of multi-layer neural networks using 2D NAND flash memory cell as a high-density and reliable synaptic device. Our operation scheme eliminates the waste of NAND flash cells and allows analogue input values. A 3-layer perceptron network with 40,545 synapses is trained on a MNIST database set using an adaptive weight update method for hardware-based multi-layer neural networks. The conductance response of NAND flash cells is measured and it is shown that the unidirectional conductance response is suitable for implementing multi-layer neural networks using NAND flash memory cells as synaptic devices. Using an online-learning, we obtained higher learning accuracy with NAND synaptic devices compared to that with a memristor-based synapse regardless of weight update methods. Using an adaptive weight update method based on a unidirectional conductance response, we obtained a 94.19% learning accuracy with NAND synaptic devices. This accuracy is comparable to 94.69% obtained by synapses based on the ideal perfect linear device. Therefore, NAND flash memory which is mature technology and has great advantage in cell density can be a promising synaptic device for implementing high-density multi-layer neural networks. INDEX TERMS Neuromorphic, NAND flash memory, deep neural networks (DNNs), synaptic device, deep learning, multi-layer neural networks, hardware-based neural network.

show abstract

“…The work in [36] experimentally shows the possibility to tune the memristive device within 1% accuracy degradation with respect to the desired state, within the dynamic range of the device. Furthermore, the usage of on-chip learning schemes will be helpful to account for these variations [50] Resistance variations can occur in the neuron memristor as well. However, this does not cause any significant read error at the output since the off to on resistance ratio is in the order of 10 4 − 10 7 [15] and the resistor divider circuit is capable of detecting this large drop with almost zero error (refer to section V).…”

Section: The Impact Of Synaptic Weight Variationsmentioning

confidence: 99%

An All-Memristor Deep Spiking Neural Computing System: A Step Toward Realizing the Low-Power Stochastic Brain

Wijesinghe

Ankit

Sengupta

et al. 2018

IEEE Trans. Emerg. Top. Comput. Intell.

View full text Add to dashboard Cite

Deep 'Analog Artificial Neural Networks' (ANNs) perform complex classification problems with remarkably high accuracy. However, they rely on humongous amount of power to perform the calculations, veiling the accuracy benefits. The biological brain on the other hand is significantly more powerful than such networks and consumes orders of magnitude less power, indicating us about some conceptual mismatch. Given that the biological neurons communicate using energy efficient trains of spikes, and the behavior is non-deterministic, incorporating these effects in Deep Artificial Neural Networks may drive us few steps towards a more realistic neuron. In this work, we propose how the inherent stochasticity of nano-scale resistive devices can be harnessed to emulate the functionality of a spiking neuron that can be incorporated in deep stochastic Spiking Neural Networks (SNN). At the algorithmic level, we propose how the training can be modified to convert an ANN to an SNN while supporting the stochastic activation function offered by these devices. We devise circuit architectures to incorporate stochastic memristive neurons along with memristive crossbars which perform the functionality of the synaptic weights. We tested the proposed All Memristor deep stochastic SNN for image classification and observed only about 1% degradation in accuracy with the ANN baseline after incorporating the circuit and device related non-idealities. We witnessed that the network is robust to certain variations and consumes ∼ 6.4× less energy than its CMOS counterpart.

show abstract

Adaptive learning rule for hardware-based deep neural networks using electronic synapse devices

Cited by 58 publications

References 37 publications

Low-Power and High-Density Neuron Device for Simultaneous Processing of Excitatory and Inhibitory Signals in Neuromorphic Systems

Low-Power and High-Density Neuron Device for Simultaneous Processing of Excitatory and Inhibitory Signals in Neuromorphic Systems

Operation Scheme of Multi-Layer Neural Networks Using NAND Flash Memory as High-Density Synaptic Devices

An All-Memristor Deep Spiking Neural Computing System: A Step Toward Realizing the Low-Power Stochastic Brain

Contact Info

Product

Resources

About