Empirical study on the efficiency of Spiking Neural Networks with axonal delays, and algorithm-hardware benchmarking

Saucedo, Alberto; Yousefzadeh, Amirreza; Tang, Guangzhi; Corradi, Federico; Linares-Barranco, B.; Sifalakis, Manolis

doi:10.1109/iscas46773.2023.10181778

Cited by 6 publications

(3 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Notably, this performance is competitive compared to these results that employ the same data processing methods and network architecture. Patiño-Saucedo et al (2023) introduce axonal delays in tandem with learnable time constants, enabling a reduction in model size to a mere 0.1 M while preserving competitive performance.…”

Section: Overall Resultsmentioning

confidence: 99%

“…These developments have spurred the exploration of jointly training synaptic weights and axonal delay in deep SNNs. While earlier research mainly centered on fixed delays with trainable weights (Bohte et al, 2002) and the concurrent training of synaptic weights and delays in shallow SNNs featuring a single layer (Taherkhani et al, 2015;Wang et al, 2019;Zhang et al, 2020), there has recently been a degree of investigation into the joint training of the synaptic weights and axonal delays in deep SNNs (Shrestha and Orchard, 2018;Shrestha et al, 2022;Sun et al, 2022Sun et al, , 2023aHammouamri et al, 2023;Patiño-Saucedo et al, 2023). Our prior effort (Sun et al, 2022) stands as one of the initial successful attempts in applying this method to deep SNNs, achieving promising results in tasks characterized by high temporal complexity.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Learnable axonal delay in spiking neural networks improves spoken word recognition

Sun,

Chua,

Devos

et al. 2023

Front. Neurosci.

View full text Add to dashboard Cite

Spiking neural networks (SNNs), which are composed of biologically plausible spiking neurons, and combined with bio-physically realistic auditory periphery models, offer a means to explore and understand human auditory processing-especially in tasks where precise timing is essential. However, because of the inherent temporal complexity in spike sequences, the performance of SNNs has remained less competitive compared to artificial neural networks (ANNs). To tackle this challenge, a fundamental research topic is the configuration of spike-timing and the exploration of more intricate architectures. In this work, we demonstrate a learnable axonal delay combined with local skip-connections yields state-of-the-art performance on challenging benchmarks for spoken word recognition. Additionally, we introduce an auxiliary loss term to further enhance accuracy and stability. Experiments on the neuromorphic speech benchmark datasets, NTIDIDIGITS and SHD, show improvements in performance when incorporating our delay module in comparison to vanilla feedforward SNNs. Specifically, with the integration of our delay module, the performance on NTIDIDIGITS and SHD improves by 14% and 18%, respectively. When paired with local skip-connections and the auxiliary loss, our approach surpasses both recurrent and convolutional neural networks, yet uses 10 × fewer parameters for NTIDIDIGITS and 7 × fewer for SHD.

show abstract

Section: Overall Resultsmentioning

confidence: 99%

mentioning

confidence: 99%

Learnable axonal delay in spiking neural networks improves spoken word recognition

Sun,

Chua,

Devos

et al. 2023

Front. Neurosci.

View full text Add to dashboard Cite

show abstract

“…A method for training per neuron axonal delays based on the SLAYER learning paradigm was proposed (Shrestha and Orchard, 2018) and extended (Sun et al, 2023b) with trainable delay caps. Recently, the effects of axonal synaptic delay learning were studied by pruning multiple delay synapses (Patiño-Saucedo et al, 2023), modeling a one-layer multinomial logistic regression with synaptic delays (Grimaldi and Perrinet, 2023) and learning delays represented trough 1D convolutions with learnable spacings (Hammouamri et al, 2023). Similarly, in order to train synaptic delays, spike trains were transformed into continuous analog, differentiable signals (Wang et al, 2019).…”

Section: Figurementioning

confidence: 99%

Co-learning synaptic delays, weights and adaptation in spiking neural networks

Deckers,

Van Damme,

Van Leekwijck

et al. 2024

Front. Neurosci.

View full text Add to dashboard Cite

Spiking neural network (SNN) distinguish themselves from artificial neural network (ANN) because of their inherent temporal processing and spike-based computations, enabling a power-efficient implementation in neuromorphic hardware. In this study, we demonstrate that data processing with spiking neurons can be enhanced by co-learning the synaptic weights with two other biologically inspired neuronal features: (1) a set of parameters describing neuronal adaptation processes and (2) synaptic propagation delays. The former allows a spiking neuron to learn how to specifically react to incoming spikes based on its past. The trained adaptation parameters result in neuronal heterogeneity, which leads to a greater variety in available spike patterns and is also found in the brain. The latter enables to learn to explicitly correlate spike trains that are temporally distanced. Synaptic delays reflect the time an action potential requires to travel from one neuron to another. We show that each of the co-learned features separately leads to an improvement over the baseline SNN and that the combination of both leads to state-of-the-art SNN results on all speech recognition datasets investigated with a simple 2-hidden layer feed-forward network. Our SNN outperforms the benchmark ANN on the neuromorphic datasets (Spiking Heidelberg Digits and Spiking Speech Commands), even with fewer trainable parameters. On the 35-class Google Speech Commands dataset, our SNN also outperforms a GRU of similar size. Our study presents brain-inspired improvements in SNN that enable them to excel over an equivalent ANN of similar size on tasks with rich temporal dynamics.

show abstract

DenRAM: neuromorphic dendritic architecture with RRAM for efficient temporal processing with delays

D’Agostino,

Moro,

Torchet

et al. 2024

Nat Commun

View full text Add to dashboard Cite

An increasing number of studies are highlighting the importance of spatial dendritic branching in pyramidal neurons in the neocortex for supporting non-linear computation through localized synaptic integration. In particular, dendritic branches play a key role in temporal signal processing and feature detection. This is accomplished thanks to coincidence detection (CD) mechanisms enabled by the presence of synaptic delays that align temporally disparate inputs for effective integration. Computational studies on spiking neural networks further highlight the significance of delays for achieving spatio-temporal pattern recognition with pure feed-forward neural networks, without the need of resorting to recurrent architectures. In this work, we present “DenRAM”, the first realization of a feed-forward spiking neural network with dendritic compartments, implemented using analog electronic circuits integrated into a 130 nm technology node and coupled with Resistive Random Access Memory (RRAM) technology. DenRAM’s dendritic circuits use RRAM devices to implement both delays and synaptic weights in the network. By configuring the RRAM devices to reproduce bio-realistic timescales, and by exploiting their heterogeneity we experimentally demonstrate DenRAM’s ability to replicate synaptic delay profiles, and to efficiently implement CD for spatio-temporal pattern recognition. To validate the architecture, we conduct comprehensive system-level simulations on two representative temporal benchmarks, demonstrating DenRAM’s resilience to analog hardware noise, and its superior accuracy compared to recurrent architectures with an equivalent number of parameters. DenRAM not only brings rich temporal processing capabilities to neuromorphic architectures, but also reduces the memory footprint of edge devices, warrants high accuracy on temporal benchmarks, and represents a significant step-forward in low-power real-time signal processing technologies.

show abstract

Empirical study on the efficiency of Spiking Neural Networks with axonal delays, and algorithm-hardware benchmarking

Cited by 6 publications

References 28 publications

Learnable axonal delay in spiking neural networks improves spoken word recognition

Learnable axonal delay in spiking neural networks improves spoken word recognition

Co-learning synaptic delays, weights and adaptation in spiking neural networks

DenRAM: neuromorphic dendritic architecture with RRAM for efficient temporal processing with delays

Contact Info

Product

Resources

About