STiDi-BP: Spike time displacement based error backpropagation in multilayer spiking neural networks

Mirsadeghi, Maryam; Shalchian, Majid; Kheradpisheh, Saeed Reza; Masquelier, Timothée

doi:10.1016/j.neucom.2020.11.052

Cited by 69 publications

(36 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As illustrated in Figure 1A, the backpropagation algorithm is one successful method for training deep SNNs (Deng et al, 2020;Taherkhani et al, 2020). Although the spike-based BP algorithm can achieve better accuracy than the STDP-based learning algorithm (Kheradpisheh et al, 2020;Rathi et al, 2020;Mirsadeghi et al, 2021), it suffers from the same fundamental disadvantage: the computation of neurons theoretically occurs at the spike neuron and requires massive data and effort. Therefore, exploring the BP algorithm for temporal encoding is more efficient for hardware implementation.…”

Section: Bp Methodsmentioning

confidence: 99%

“…As described in Figure 9, the proposed SSTDP method reduces the computation by orders of magnitude over the SNN with ratebased coding in Tavanaei et al (2019) and will have an advantage over time-based SNN. Unlike our SSTDP method, other timebased approaches (Kheradpisheh et al, 2020;Mirsadeghi et al, 2021) force all neurons to fire spikes, even those that have not fired, to improve the network performance, and such an approach increases the computation effort. In addition, the network scale used in the two compared baselines is also larger than ours but with less accuracy than our method, as illustrated in the table.…”

Section: Effect Of Computation Costmentioning

confidence: 99%

“…There have been some successful attempts to introduce the BP-based learning mechanisms into SNNs (Lee et al, 2016;Tavanaei et al, 2019;Zhou et al, 2019;Kheradpisheh et al, 2020;Fang et al, 2021;Mirsadeghi et al, 2021). The first approach is the spike-based BP, which treats the membrane potentials as differentiable activations of spiking neurons and trains the synaptics of SNNs in a layer-wise fashion (Tavanaei et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

“…Although these two types of methods are suitable for gradient descent learning, it requires complicated procedures for computing the derivative of the loss function in spatial and temporal domains. The third approach is approximate methods, which estimate the surrogate gradient of the spike generation function (Mirsadeghi et al, 2021). However, this kind of approach incurs the strong assumption in backpropagating the error through the network using the chain rule.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

SSTDP: Supervised Spike Timing Dependent Plasticity for Efficient Spiking Neural Network Training

et al. 2021

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) are a pathway that could potentially empower low-power event-driven neuromorphic hardware due to their spatio-temporal information processing capability and high biological plausibility. Although SNNs are currently more efficient than artificial neural networks (ANNs), they are not as accurate as ANNs. Error backpropagation is the most common method for directly training neural networks, promoting the prosperity of ANNs in various deep learning fields. However, since the signals transmitted in the SNN are non-differentiable discrete binary spike events, the activation function in the form of spikes presents difficulties for the gradient-based optimization algorithms to be directly applied in SNNs, leading to a performance gap (i.e., accuracy and latency) between SNNs and ANNs. This paper introduces a new learning algorithm, called SSTDP, which bridges the gap between backpropagation (BP)-based learning and spike-time-dependent plasticity (STDP)-based learning to train SNNs efficiently. The scheme incorporates the global optimization process from BP and the efficient weight update derived from STDP. It not only avoids the non-differentiable derivation in the BP process but also utilizes the local feature extraction property of STDP. Consequently, our method can lower the possibility of vanishing spikes in BP training and reduce the number of time steps to reduce network latency. In SSTDP, we employ temporal-based coding and use Integrate-and-Fire (IF) neuron as the neuron model to provide considerable computational benefits. Our experiments show the effectiveness of the proposed SSTDP learning algorithm on the SNN by achieving the best classification accuracy 99.3% on the Caltech 101 dataset, 98.1% on the MNIST dataset, and 91.3% on the CIFAR-10 dataset compared to other SNNs trained with other learning methods. It also surpasses the best inference accuracy of the directly trained SNN with 25~32× less inference latency. Moreover, we analyze event-based computations to demonstrate the efficacy of the SNN for inference operation in the spiking domain, and SSTDP methods can achieve 1.3~37.7× fewer addition operations per inference. The code is available at: https://github.com/MXHX7199/SNN-SSTDP.

show abstract

Section: Bp Methodsmentioning

confidence: 99%

Section: Effect Of Computation Costmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

SSTDP: Supervised Spike Timing Dependent Plasticity for Efficient Spiking Neural Network Training

et al. 2021

View full text Add to dashboard Cite

show abstract

“…BP neural network is widely used in all walks of life because of its strong adaptability, including nonlinear mapping ability, self-learning ability, adaptive ability, generalization ability, fault tolerance ability, and other advantages [39,42,43]. At the same time, many scholars have improved the BP neural network considering its shortcomings and deficiencies, which improves the accuracy of the model [44][45][46].…”

Section: Mathematical Problems In Engineeringmentioning

confidence: 99%

Probabilistic Prediction of Unsafe Event in Air Traffic Control Department Based on the Improved Backpropagation Neural Network

Liao

Miao

Yang

2021

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Air traffic control is an important tool to ensure the safety of civil aviation. For the departments that do the work of air traffic control, reducing the percentage of unsafe event is the core task of safety management. If the relationship between the percentage of unsafe event and their influencing factors can be effectively clarified, then the probability of unsafe event in some control department can be predicted. So, it is of great importance to improve the level of safety management. To quantitatively estimate the probability of unsafe event, a three-layer BP neural network model is introduced in this paper. First, a probabilistic representation of unsafe event related to air traffic control department is made, and then, the probability of different classes of unsafe events and safe events is taken as the outputs of the BP neural network, the factors influencing occurrence of unsafe event connected with air traffic control is taken as inputs, and the sigmoid function is chosen as activation function for the hidden layer. Based on the error function of neural network, it is proved that the general BP neural network has two drawbacks when used for the training of small probability events, which are as follows: the pattern does not ensure that the sum of probability of all events is equal to one and the relative error between the actual outputs and desired outputs is very large after the training of neural network. The reason proved in this paper is that the occurrence rate of the unsafe event is much smaller than that of the safe event, resulting in each weight in the hide layer being subjected to the desired outputs of the safe event when using the gradient descent method for network training. To address this issue, a new mapping method is put forward to reduce the large difference of the desired outputs between the safe event and unsafe event. It is theoretically proved that the mapping method proposed in this paper can not only improve the training accuracy but also ensure that the sum of probability is equal to one. Finally, a numeric example is given to demonstrate that the method proposed in this paper is effective and feasible.

show abstract

Recurrent spiking neural network with dynamic presynaptic currents based on backpropagation

Wang

Zhang

Shi

et al. 2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

In recent years, spiking neural networks (SNNs), which originated from the theoretical basis of neuroscience, have attracted neuromorphic computing and brain‐like computing due to their advantages, such as neural dynamics and coding mechanism, which are similar to biological neurons. SNNs have become one of the mainstream frameworks in the field of brain‐like computing. However, most of the Leaky Integrate‐and‐Fire (LIF) neuron models currently used by SNNs based on direct training of backpropagation (BP) do not consider the changes in the recurrent connections and the dynamic strength of neuron connections over time. This study presented the LIF neuron model with recurrent connections and a method for dynamically changing the presynaptic currents. Recurrent LIF neurons have an additional cyclic connection compared with classic LIF neurons. Their postsynaptic current stimulates a change in membrane potential at the next time point. Their dynamics were more similar to the activities of biological neurons. We also proposed an efficient and flexible BP training method for recurrent LIF neurons. On the basis of the above methods, we proposed the recurrent SNN with dynamic presynaptic currents based on backpropagation (RDS‐BP). We test the proposed RDS‐BP on three image data sets (MNIST, Fashion‐MNIST and CIFAR‐10) and two text data sets (IMDB and TREC). The results showed that the performance of RDS‐BP not only exceeded the naive SNN models based on BP but also exceeded the SNN methods proposed in previous studies in recent years, which had excellent performance in previous experiments. Our work provides a new LIF neuron model with a recurrent connection and dynamic presynaptic current and a BP training arrangement for the proposed neuron, which could merit developments with neuromorphic and brain‐like computing.

show abstract

STiDi-BP: Spike time displacement based error backpropagation in multilayer spiking neural networks

Cited by 69 publications

References 38 publications

SSTDP: Supervised Spike Timing Dependent Plasticity for Efficient Spiking Neural Network Training

SSTDP: Supervised Spike Timing Dependent Plasticity for Efficient Spiking Neural Network Training

Probabilistic Prediction of Unsafe Event in Air Traffic Control Department Based on the Improved Backpropagation Neural Network

Recurrent spiking neural network with dynamic presynaptic currents based on backpropagation

Contact Info

Product

Resources

About