Axonal Delay as a Short-Term Memory for Feed Forward Deep Spiking Neural Networks

Sun, Pengfei; Zhu, Longwei; Botteldooren, Dick

doi:10.1109/icassp43922.2022.9747411

Cited by 12 publications

(7 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We will follow the shorthand notation in [15] to represent the architecture: layers are separated by -, spatial dimensions are separated by x, an N × N convolution filter with K channels is represented by KcN, an N ×N aggregate pooling filter is represented by Na and a dense layer with N neurons is represented by the number itself. Note that the convolution and dense layer neurons also have trainable axonal delays [37]. In all our experiments, we use neuron threshold ϑ = 10 mv, and sampling time of 1 ms.…”

Section: Experiments and Resultsmentioning

confidence: 99%

Spikemax: Spike-based Loss Methods for Classification

Shrestha¹,

Zhu²,

Sun³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) are a promising research paradigm for low power edge-based computing. Recent works in SNN backpropagation has enabled training of SNNs for practical tasks. However, since spikes are binary events in time, standard loss formulations are not directly compatible with spike output. As a result, current works are limited to using meansquared loss of spike count. In this paper, we formulate the output probability interpretation from the spike count measure and introduce spike-based negative log-likelihood measure which are more suited for classification tasks especially in terms of the energy efficiency and inference latency. We compare our loss measures with other existing alternatives and evaluate using classification performances on three neuromorphic benchmark datasets: NMNIST, DVS Gesture and N-TIDIGITS18. In addition, we demonstrate state of the art performances on these datasets, achieving faster inference speed and less energy consumption.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

Spikemax: Spike-based Loss Methods for Classification

Shrestha¹,

Zhu²,

Sun³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Delays in SNN Many methods have been proposed for adapting propagation delays, inspired by spike timing dependent plasticity (Wang et al, 2013) or based on the ReSuMe learning rule (Zhang et al, 2020). A method for training per neuron axonal delays based on the SLAYER learning paradigm was proposed (Shrestha and Orchard, 2018) and extended (Sun et al, 2023b) with trainable delay caps. Recently, the effects of axonal synaptic delay learning were studied by pruning multiple delay synapses (Patiño-Saucedo et al, 2023), modeling a one-layer multinomial logistic regression with synaptic delays (Grimaldi and Perrinet, 2023) and learning delays represented trough 1D convolutions with learnable spacings (Hammouamri et al, 2023).…”

Section: Figurementioning

confidence: 99%

Co-learning synaptic delays, weights and adaptation in spiking neural networks

Deckers,

Van Damme,

Van Leekwijck

et al. 2024

Front. Neurosci.

View full text Add to dashboard Cite

Spiking neural network (SNN) distinguish themselves from artificial neural network (ANN) because of their inherent temporal processing and spike-based computations, enabling a power-efficient implementation in neuromorphic hardware. In this study, we demonstrate that data processing with spiking neurons can be enhanced by co-learning the synaptic weights with two other biologically inspired neuronal features: (1) a set of parameters describing neuronal adaptation processes and (2) synaptic propagation delays. The former allows a spiking neuron to learn how to specifically react to incoming spikes based on its past. The trained adaptation parameters result in neuronal heterogeneity, which leads to a greater variety in available spike patterns and is also found in the brain. The latter enables to learn to explicitly correlate spike trains that are temporally distanced. Synaptic delays reflect the time an action potential requires to travel from one neuron to another. We show that each of the co-learned features separately leads to an improvement over the baseline SNN and that the combination of both leads to state-of-the-art SNN results on all speech recognition datasets investigated with a simple 2-hidden layer feed-forward network. Our SNN outperforms the benchmark ANN on the neuromorphic datasets (Spiking Heidelberg Digits and Spiking Speech Commands), even with fewer trainable parameters. On the 35-class Google Speech Commands dataset, our SNN also outperforms a GRU of similar size. Our study presents brain-inspired improvements in SNN that enable them to excel over an equivalent ANN of similar size on tasks with rich temporal dynamics.

show abstract

“…Shrestha and Orchard ( 2018 ) proposes a general backpropagation mechanism for learning synaptic weights and axonal delays which overcomes the problem of non-differentiability of the spike function and uses a temporal credit assignment policy for backpropagating error to preceding layers. Sun et al ( 2022 ) proposes the rectified axonal delay (RAD) as an additional degree of freedom for training that can easily be incorporated into existing SNN frameworks. The new model can perform well on problems where timing matters using very few parameters.…”

Section: Introductionmentioning

confidence: 99%

Training multi-layer spiking neural networks with plastic synaptic weights and delays

Wang

2024

Front. Neurosci.

View full text Add to dashboard Cite

Spiking neural networks are usually considered as the third generation of neural networks, which hold the potential of ultra-low power consumption on corresponding hardware platforms and are very suitable for temporal information processing. However, how to efficiently train the spiking neural networks remains an open question, and most existing learning methods only consider the plasticity of synaptic weights. In this paper, we proposed a new supervised learning algorithm for multiple-layer spiking neural networks based on the typical SpikeProp method. In the proposed method, both the synaptic weights and delays are considered as adjustable parameters to improve both the biological plausibility and the learning performance. In addition, the proposed method inherits the advantages of SpikeProp, which can make full use of the temporal information of spikes. Various experiments are conducted to verify the performance of the proposed method, and the results demonstrate that the proposed method achieves a competitive learning performance compared with the existing related works. Finally, the differences between the proposed method and the existing mainstream multi-layer training algorithms are discussed.

show abstract

Axonal Delay as a Short-Term Memory for Feed Forward Deep Spiking Neural Networks

Cited by 12 publications

References 23 publications

Spikemax: Spike-based Loss Methods for Classification

Spikemax: Spike-based Loss Methods for Classification

Co-learning synaptic delays, weights and adaptation in spiking neural networks

Training multi-layer spiking neural networks with plastic synaptic weights and delays

Contact Info

Product

Resources

About