Spiking Neural Network (SNN) With Memristor Synapses Having Non-linear Weight Update

Kim, Taeyoon; Hu, Suman; Kim, Jaewook; Kwak, Joon Young; Park, Jongkil; Lee, Jun Young; Kim, Inho; Park, Jong‐Keuk; Jeong, YeonJoo

doi:10.3389/fncom.2021.646125

Cited by 52 publications

(31 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The main figure-of-merit of a memristor enabling its use as electronic synapse in a SNN is the spike-timing dependent plasticity (STDP) plot, a temporally asymmetric form of Hebbian learning that is induced by tight temporal correlations between the spikes of pre-and postsynaptic neurons 13 . A few implementations of SNN with memristors have been reported although a strong software perspective is employed in the studies in most of the cases 14,15 . Thereby both the synapses and neuron realisations have to be investigated deeply in the future.…”

Section: Introductionmentioning

confidence: 99%

Spiking neural networks based on two-dimensional materials

et al. 2022

View full text Add to dashboard Cite

The development of artificial neural networks using memristors is gaining a lot of interest among technological companies because it can reduce the computing time and energy consumption. There is still no memristor, made of any material, capable to provide the ideal figures-of-merit required for the implementation of artificial neural networks, meaning that more research is required. Here we present the use of multilayer hexagonal boron nitride based memristors to implement spiking neural networks for image classification. Our study indicates that the recognition accuracy of the network is high, and that can be resilient to device variability if the number of neurons employed is large enough. There are very few studies that present the use of a two-dimensional material for the implementation of synapses of different features; in our case, in addition to a study of the synaptic characteristics of our memristive devices, we deal with complete spiking neural network training and inference processes.

show abstract

Section: Introductionmentioning

confidence: 99%

Spiking neural networks based on two-dimensional materials

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Accordingly, the update asymmetry of devices is more likely to affect the process of training neural networks (Kwon et al, 2020 ). Inspired by this motivation, they demonstrated that asymmetric non-linear devices are more powerful than symmetric linear devices (Brivio et al, 2018 , 2021 ; Kim et al, 2021 ). Our analytical tools and calibration method can be applied to illustrate the relationship between the update asymmetry of devices and its impact on training in detail.…”

Section: Impact Of Update Asymmetries On Neural Network Performancementioning

confidence: 99%

Impact of Asymmetric Weight Update on Neural Network Training With Tiki-Taka Algorithm

Lee

Noh

et al. 2022

Front. Neurosci.

View full text Add to dashboard Cite

Recent progress in novel non-volatile memory-based synaptic device technologies and their feasibility for matrix-vector multiplication (MVM) has ignited active research on implementing analog neural network training accelerators with resistive crosspoint arrays. While significant performance boost as well as area- and power-efficiency is theoretically predicted, the realization of such analog accelerators is largely limited by non-ideal switching characteristics of crosspoint elements. One of the most performance-limiting non-idealities is the conductance update asymmetry which is known to distort the actual weight change values away from the calculation by error back-propagation and, therefore, significantly deteriorates the neural network training performance. To address this issue by an algorithmic remedy, Tiki-Taka algorithm was proposed and shown to be effective for neural network training with asymmetric devices. However, a systematic analysis to reveal the required asymmetry specification to guarantee the neural network performance has been unexplored. Here, we quantitatively analyze the impact of update asymmetry on the neural network training performance when trained with Tiki-Taka algorithm by exploring the space of asymmetry and hyper-parameters and measuring the classification accuracy. We discover that the update asymmetry level of the auxiliary array affects the way the optimizer takes the importance of previous gradients, whereas that of main array affects the frequency of accepting those gradients. We propose a novel calibration method to find the optimal operating point in terms of device and network parameters. By searching over the hyper-parameter space of Tiki-Taka algorithm using interpolation and Gaussian filtering, we find the optimal hyper-parameters efficiently and reveal the optimal range of asymmetry, namely the asymmetry specification. Finally, we show that the analysis and calibration method be applicable to spiking neural networks.

show abstract

“…Different types of memristive devices are explored for their use in neuromorphic architectures. Among the most prominent ones are Phase Change Memory (PCM) (Boybat et al (2018a); Mehonic et al (2020)), Spin-Transfer Torque Magnetoresistive Random Access Memory (STT-MRAM) (Jung et al (2022); Ham et al (2021)) and Redox-based Random Access Memories (ReRAM) (Kim et al, 2021); Ziegler et al (2020); Bengel et al (2021a); Covi et al (2016); Park et al (2012)). For redox based resistive switches one can further differentiate between electrochemical metallization mechanism (ECM) cells and valence change mechanism (VCM) cells.…”

Section: Introductionmentioning

confidence: 99%

Tailor-made synaptic dynamics based on memristive devices

Bengel

Zhang

Mohr

et al. 2023

Front. Electron. Mater

View full text Add to dashboard Cite

The proliferation of machine learning algorithms in everyday applications such as image recognition or language translation has increased the pressure to adapt underlying computing architectures towards these algorithms. Application specific integrated circuits (ASICs) such as the Tensor Processing Units by Google, Hanguang by Alibaba or Inferentia by Amazon Web Services were designed specifically for machine learning algorithms and have been able to outperform CPU based solutions by great margins during training and inference. As newer generations of chips allow handling of and computation on more and more data, the size of neural networks has dramatically increased, while the challenges they are trying to solve have become more complex. Neuromorphic computing tries to take inspiration from biological information processing systems, aiming to further improve the efficiency with which these networks can be trained or the inference can be performed. Enhancing neuromorphic computing architectures with memristive devices as non-volatile storage elements could potentially allow for even higher energy efficiencies. Their ability to mimic synaptic plasticity dynamics brings neuromorphic architectures closer to the biological role models. So far, memristive devices are mainly investigated for the emulation of the weights of neural networks during training and inference as their non-volatility would enable both processes in the same location without data transfer. In this paper, we explore realisations of different synapses build from memristive ReRAM devices, based on the Valence Change Mechanism. These synapses are the 1R synapse, the NR synapse and the 1T1R synapse. For the 1R synapse, we propose three dynamical regimes and explore their performance through different synapse criteria. For the NR synapse, we discuss how the same dynamical regimes can be addressed in a more reliable way. We also show experimental results measured on ZrOx devices to support our simulation based claims. For the 1T1R synapse, we explore the trade offs between the connection direction of the ReRAM device and the transistor. For all three synapse concepts we discuss the impact of device-to-device and cycle-to-cycle variability. Additionally, the impact of the stimulation mode on the observed behavior is discussed.

show abstract

Spiking Neural Network (SNN) With Memristor Synapses Having Non-linear Weight Update

Cited by 52 publications

References 37 publications

Spiking neural networks based on two-dimensional materials

Spiking neural networks based on two-dimensional materials

Impact of Asymmetric Weight Update on Neural Network Training With Tiki-Taka Algorithm

Tailor-made synaptic dynamics based on memristive devices

Contact Info

Product

Resources

About