Training Spiking Neural Networks Using Lessons From Deep Learning

Eshraghian, Jason K.; Ward, Max; Neftci, Emre O.; Wang, Xinxin; Lenz, Gregor; Dwivedi, Girish; Bennamoun, Mohammed; Jeong, Doo Seok; Lu, Wei D.

doi:10.1109/jproc.2023.3308088

Cited by 215 publications

(47 citation statements)

References 210 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, a fully‐connected three‐layer SNN have been stimulated by utilizing the snnTorch platform, as shown in Figure a. [ 30,31 ] The spiking neuron can be described by a LIF neuronal model based on the experimental data in Figure 3c,d, where the firing rate has been fitted for training SNN, as exhibited in Figure 5b,c respectively. The rate of spikes sent to the input layer are proportional to the grayscale values of each pixel in the identified image, referring to Figure S10 (Supporting Information) for details.…”

Section: Resultsmentioning

confidence: 99%

Artificial Neurons Using Ag−In−Zn−S/Sericin Peptide‐Based Threshold Switching Memristors for Spiking Neural Networks

He,

Yan,

Zhang

et al. 2023

Adv Elect Materials

View full text Add to dashboard Cite

Memristive devices with threshold switching characteristics can be effectively utilized to mimic biological neurons acting as one of the key building blocks for constructing advanced hardware neural networks. In this work, the emulation of leaky integrate‐and‐fire memristive neuron is realized in one single cell with Ag/Ag−In−Zn−S/silk sericin/W architecture without the need for additional auxiliary circuits. The studied devices demonstrate excellent electrical properties, such as stably repeatable threshold switching, concentratedly low threshold voltage (≈0.4 V), and relatively small device‐to‐device variation. In addition, multiple neural features, such as leaky integrate‐and‐fire neuron functionality and strength‐modulated spike frequency characteristic, have been successfully emulated owing to the forming‐free volatile threshold switching effect. The stable volatile threshold switching behaviors and regular firing event may be attributed to the controllable metallic Ag filamentary mechanism. Furthermore, a solid accuracy of 91.44% of the pattern recognition of Modified National Institute of Standards and Technology (MNIST) data is obtained via a trained spiking neural network (SNN) based on the leaky integrate‐and‐fire behavior of sericin‐based device. These achievements shed light on the fact that employing sericin biomaterials has great application potential in advanced neuromorphic computation.

show abstract

Section: Resultsmentioning

confidence: 99%

Artificial Neurons Using Ag−In−Zn−S/Sericin Peptide‐Based Threshold Switching Memristors for Spiking Neural Networks

He,

Yan,

Zhang

et al. 2023

Adv Elect Materials

View full text Add to dashboard Cite

show abstract

“…Another common training method uses the surrogate gradient learning algorithm, as described in the study of Eshraghian et al [ 32 ] This approach leverages the sigmoid function gradient instead of the actual neuron gradient. The weight update rules for this algorithm can be expressed as

\omega_{i j}^{l} = \omega_{i j}^{l} - \alpha \frac{\partial E}{\partial \omega_{i j}^{l}} , \textrm{ } \textrm{ } \text{where} \textrm{ } \textrm{ } \frac{\partial E}{\partial \omega_{i j}^{l}} = \frac{\partial E}{\partial S} \frac{\partial S}{\partial V} \frac{\partial V}{\partial I} \frac{\partial I}{\partial \omega_{i j}^{l}}

where S is the spike operator function, which is nondifferentiable and cannot be used for gradient descent optimization.…”

Section: Methodsmentioning

confidence: 99%

Enhancing Robustness of Memristor Crossbar‐Based Spiking Neural Networks against Nonidealities: A Hybrid Approach for Neuromorphic Computing in Noisy Environments

Zhang,

Sun,

Xie

et al. 2023

Advanced Intelligent Systems

View full text Add to dashboard Cite

Memristor crossbar‐based spiking neural networks (SNNs) face challenges caused by nonidealities associated with their hardware‐based neurons and synapses. The key nonidealities include electric‐field noise, conductance noise, and conductance drift. This study investigates the robustness of fully connected, convolutional, residual, and spike‐timing‐dependent plasticity‐based SNNs against hardware nonidealities using the MNIST, Fashion MNIST, and CIFAR10 datasets. In response to these challenges, a novel hybrid residual SNN (HRSNN) is proposed that incorporates a new neuron circuit and a weight‐dependent loss function. The HRSNN in a high‐intensity noise environment is evaluated using the neuromorphic DVS128 Gesture dataset. The achieved accuracy rate of 92.71% is only 2.15% lower than that of the noise‐free environment. These results demonstrate the robustness of the proposed HRSNN under high‐intensity noise conditions and present new possibilities for the advancement of neuromorphic computing in noisy environments.

show abstract

“…Neuromorphic Hardware Simulation: The simulation was carried out in Python 3 and based on the snnTorch framework. [37] The autoencoder was based on the encoder-code-decoder structure. The public dataset for anomaly detection was from the National Aeronautics and Space Administration (NASA) Mars Science Laboratory (MSL) (available at https:// github.com/NetManAIOps/OmniAnomaly).…”

Section: Methodsmentioning

confidence: 99%

Spiking Neurons with Neural Dynamics Implemented Using Stochastic Memristors

Song,

Liu,

Pei

et al. 2023

Adv Elect Materials

View full text Add to dashboard Cite

Implementing and integrating spiking neurons for neuromorphic hardware realization conforming to spiking neural networks holds great promise in enabling efficient learning and decision‐making. The spiking neurons, however, may lack the spiking dynamics to encode the dynamical information in complex real‐world problems. Herein, using filamentary memristors from solution‐processed hexagonal boron nitride, this study assembles leaky integrate‐and‐fire spiking neurons and, particularly, harnesses the common switching stochasticity feature in the memristors to allow key neural dynamics, including Poisson‐like spiking and adaptation. The neurons, with the dynamics fitted via hardware‐algorithm codesign, suggest a potential in realizing spike‐based neuromorphic hardware capable of handling complex problems. Simulation of an autoencoder for anomaly detection of time‐series real analog and digital data from physical systems is demonstrated, underscoring its promising prospect in applications, especially, at the edges with limited computation resources, for instance, auto‐pilot, manufacturing, wearables, and Internet of things.

show abstract

Training Spiking Neural Networks Using Lessons From Deep Learning

Cited by 215 publications

References 210 publications

Artificial Neurons Using Ag−In−Zn−S/Sericin Peptide‐Based Threshold Switching Memristors for Spiking Neural Networks

Artificial Neurons Using Ag−In−Zn−S/Sericin Peptide‐Based Threshold Switching Memristors for Spiking Neural Networks

Enhancing Robustness of Memristor Crossbar‐Based Spiking Neural Networks against Nonidealities: A Hybrid Approach for Neuromorphic Computing in Noisy Environments

Spiking Neurons with Neural Dynamics Implemented Using Stochastic Memristors

Contact Info

Product

Resources

About