Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks

Wu, Yujie; Deng, Lei; Li, Guoqi; Zhu, Jun; Shi, Luping

doi:10.3389/fnins.2018.00331

Cited by 767 publications

(606 citation statements)

References 47 publications

Supporting

Mentioning

559

Contrasting

Order By: Relevance

“…Hence, the development of an efficient training algorithm for SNNs is of considerable importance. Much effort has been expended in the past two decades on this issue [10], with the subsequently developed approaches generally characterized as indirect supervised learning (SL), direct SL, or plasticity-based training [10,11]. For the indirect SL method, ANNs are first trained and then mapped to equivalent SNNs by different conversion algorithms that transform real-valued computing into spike-based computing [12,13,14,15,16,17,18,19]; however, this method does not incorporate SNN learning and therefore provides no heuristic information on how to train a SNN.…”

Section: Introductionmentioning

confidence: 99%

“…For the indirect SL method, ANNs are first trained and then mapped to equivalent SNNs by different conversion algorithms that transform real-valued computing into spike-based computing [12,13,14,15,16,17,18,19]; however, this method does not incorporate SNN learning and therefore provides no heuristic information on how to train a SNN. The direct SL method is based on the BP algorithm [11,20,21,22,23], e.g., using membrane potentials as continuous variables for calculating errors in BP [20,23] or using continuous activity function to approximate neuronal spike activity and obtain differentiable activ-ity for the BP algorithm [11,22]. However, such research must still perform numerous real-valued computations and non-local communications during the training process; thus, BP-based methods are as potentially energy inefficient as ANNs and also lack bio-plausibility.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A biologically plausible supervised learning method for spiking neural networks using the symmetric STDP rule

et al. 2020

View full text Add to dashboard Cite

Spiking neural networks (SNNs) possess energy-efficient potential due to eventbased computation. However, supervised training of SNNs remains a challenge as spike activities are non-differentiable. Previous SNNs training methods can be generally categorized into two basic classes, i.e., backpropagation-like training methods and plasticity-based learning methods. The former methods are dependent on energy-inefficient real-valued computation and non-local transmission, as also required in artificial neural networks (ANNs), whereas the latter are either considered to be biologically implausible or exhibit poor performance.Hence, biologically plausible (bio-plausible) high-performance supervised learning (SL) methods for SNNs remain deficient. In this paper, we proposed a novel bio-plausible SNN model for SL based on the symmetric spike-timing dependent plasticity (sym-STDP) rule found in neuroscience. By combining the sym-STDP rule with bio-plausible synaptic scaling and intrinsic plasticity of the dynamic threshold, our SNN model implemented SL well and achieved good performance in the benchmark recognition task (MNIST dataset). To reveal the underlying mechanism of our SL model, we visualized both layer-based * These two authors contributed equally to this work.activities and synaptic weights using the t-distributed stochastic neighbor embedding (t-SNE) method after training and found that they were well clustered, thereby demonstrating excellent classification ability. Furthermore, to verify the robustness of our model, we trained it on another more realistic dataset (Fashion-MNIST), which also showed good performance. As the learning rules were bio-plausible and based purely on local spike events, our model could be easily applied to neuromorphic hardware for online training and may be helpful for understanding SL information processing at the synaptic level in biological neural systems.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A biologically plausible supervised learning method for spiking neural networks using the symmetric STDP rule

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Most of the classification performances available in literature for SNNs are for MNIST and CIFAR-10 datasets. The popular methods for SNN training are 'Spike Time Dependent Plasticity (STDP)' based unsupervised learning [7,49,3,42,43] and 'spike-based backpropagation' based supervised learning [24,16,48,30,29]. There are a few works [45,17,46,22] which tried to combine the two approaches to get the best of both worlds.…”

Section: The Classification Performancementioning

confidence: 99%

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

et al. 2020

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) have recently emerged as a prominent neural computing paradigm. However, the typical shallow spiking network architectures have limited capacity for expressing complex representations, while training a very deep spiking network has not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-line trained deep Artificial Neural Networks (ANNs) to SNNs. However, ANN-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous and non-differentiable nature of spike generation function. To overcome this problem, we propose an approximate derivative method that accounts for leaky behavior of LIF neuron. This method enables training of deep convolutional SNNs with input spike events using spike-based backpropagation algorithm. Our experiments show the effectiveness of the proposed spike-based learning strategy on state-of-the-art deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN and CIFAR-10 datasets compared to other SNNs trained with spike-based learning. Moreover, we analyze sparse event-based computations to demonstrate the efficacy of the proposed SNN training method for inference operation in the spiking domain. and show remarkable results, which occasionally outperform human-level performance [20,13,40]. To that effect, deploying deep learning is becoming necessary not only on large-scale computers, but also on edge devices (e.g. phone, tablet, smart watch, robot etc.). However, the ever-growing complexity of the state-of-the-art deep neural networks together with the explosion in the amount of data to be processed, place significant energy demands on current computing platforms. For example, a deep ANN model requires unprecedented amount of computing hardware resources that often requires huge computing power of cloud servers and significant amount of time to train.Spiking Neural Network (SNN) is one of the leading candidates for overcoming the constraints of neural computing and to efficiently harness the machine learning algorithm in real-life (or mobile) applications [28,5]. The concepts of SNN, which is often regarded as the 3 rd generation neural network [27], are inspired by biologically plausible Leaky Integrate and Fire (LIF) spiking neuron models [6] that can efficiently process spatio-temporal information. The LIF neuron model is characterized by the internal state, called membrane potential, that integrates the inputs over time and generates an output spike whenever it overcomes the neuronal firing threshold. This mechanism enables event-based and asynchronous computations across the layers on spiking systems, which makes it naturally suitable for ultra-low power computing. Furthermore, recent works [38,35] have shown that these properties make SNNs significantly more attractive for deeper networks in the case of h...

show abstract

“…Network Architecture Method Test Accuracy (%) O'Connor (2016) [12] MLP Fractional stochastic gradient descent 97.93 Lee (2017) [8] MLP Backpropagation 98.88 Neftci (2017) [26] MLP Event-driven random backpropagation 97.98 Mostafa (2017) [11] MLP Backpropagation with temporal coding 98.00 Wu (2018) [27] MLP Spatio-Temporal Backpropagation 98.48 Diehl (2015) [17] MLP Conversion of ANNs 98.60 Neil (2016) [ In contrast to the indirect ANN to SNN conversion approach, the proposed learning rule can integrate the inference latency, spike rate and hardware constraints more effectively during the training. Hence, it allows direct deployment to neuromorphic hardware for efficient inference.…”

Section: Modelmentioning

confidence: 99%

Deep Spiking Neural Network with Spike Count based Learning Rule

Chua

Zhang

et al. 2019

2019 International Joint Conference on Neural Networks (IJCNN)

Self Cite

View full text Add to dashboard Cite

Deep spiking neural networks (SNNs) support asynchronous event-driven computation, massive parallelism and demonstrate great potential to improve the energy efficiency of its synchronous analog counterpart. However, insufficient attention has been paid to neural encoding when designing SNN learning rules. Remarkably, the temporal credit assignment has been performed on rate-coded spiking inputs, leading to poor learning efficiency. In this paper, we introduce a novel spike-based learning rule for rate-coded deep SNNs, whereby the spike count of each neuron is used as a surrogate for gradient backpropagation. We evaluate the proposed learning rule by training deep spiking multi-layer perceptron (MLP) and spiking convolutional neural network (CNN) on the UCI machine learning and MNIST handwritten digit datasets. We show that the proposed learning rule achieves state-of-the-art accuracies on all benchmark datasets. The proposed learning rule allows introducing latency, spike rate and hardware constraints into the SNN learning, which is superior to the indirect approach in which conventional artificial neural networks are first trained and then converted to SNNs. Hence, it allows direct deployment to the neuromorphic hardware and supports efficient inference. Notably, a test accuracy of 98.40% was achieved on the MNIST dataset in our experiments with only 10 simulation time steps, when the same latency constraint is imposed during training.

show abstract

Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks

Cited by 767 publications

References 47 publications

A biologically plausible supervised learning method for spiking neural networks using the symmetric STDP rule

A biologically plausible supervised learning method for spiking neural networks using the symmetric STDP rule

Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures

Deep Spiking Neural Network with Spike Count based Learning Rule

Contact Info

Product

Resources

About