Temporal Coding in Spiking Neural Networks With Alpha Synaptic Function: Learning With Backpropagation

Comşa, Iulia-Maria; Potempa, Krzysztof; Versari, Luca; Fischbacher, Thomas; Gesmundo, Andréa; Alakuijala, Jyrki

doi:10.1109/tnnls.2021.3071976

Cited by 45 publications

(27 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite the efforts to improve the efficiency of TTFS coding in deep SNNs [17,36], their improvements have been restricted by the conversionbased training algorithms. In several studies, SNNs were directly trained, but their methods were not validated as applicable to deep SNNs [12,37,38]. A recent study suggested direct training methods…”

Section: Training Methods Of Deep Snnsmentioning

confidence: 99%

T2FSNN: Deep Spiking Neural Networks with Time-to-first-spike Coding

Park

Kim

et al. 2020

2020 57th ACM/IEEE Design Automation Conference (DAC)

View full text Add to dashboard Cite

The tremendous energy consumption of deep neural networks (DNNs) has become a serious problem in deep learning. Spiking neural networks (SNNs), which mimic the operations in the human brain, have been studied as prominent energy-efficient neural networks. Due to their event-driven and spatiotemporally sparse operations, SNNs show possibilities for energy-efficient processing. To unlock their potential, deep SNNs have adopted temporal coding such as time-to-first-spike (TTFS) coding, which represents the information between neurons by the first spike time. With TTFS coding, each neuron generates one spike at most, which leads to a significant improvement in energy efficiency. Several studies have successfully introduced TTFS coding in deep SNNs, but they showed restricted efficiency improvement owing to the lack of consideration for efficiency during training. To address the aforementioned issue, this paper presents training methods for energyefficient deep SNNs with TTFS coding. We introduce a surrogate DNN model to train the deep SNN in a feasible time and analyze the effect of the temporal kernel on training performance and efficiency. Based on the investigation, we propose stochastically relaxed activation and initial value-based regularization for the temporal kernel parameters. In addition, to reduce the number of spikes even further, we present temporal kernel-aware batch normalization. With the proposed methods, we could achieve comparable training results with significantly reduced spikes, which could lead to energy-efficient deep SNNs.

show abstract

Section: Training Methods Of Deep Snnsmentioning

confidence: 99%

T2FSNN: Deep Spiking Neural Networks with Time-to-first-spike Coding

Park

Kim

et al. 2020

2020 57th ACM/IEEE Design Automation Conference (DAC)

View full text Add to dashboard Cite

show abstract

“…We use a neuronal model previously described by Comşa et al (2021). Upon spiking, an input neuron, indexed by i, produces an increase over time t in the temporal membrane of a downstream (output) neuron described by an α function (Sterratt et al, 2018) of the form w i (t − t i )e −τ (t−t i ) , where:…”

Section: Spiking Neuron Modelmentioning

confidence: 99%

“…Given a set of input spikes t i∈{1..n} and their corresponding weights w i∈{1..n} , the output spike time is given by (refer to Comşa et al, 2021 for the full derivation):…”

Section: Spiking Neuron Modelmentioning

confidence: 99%

“…Next, to backpropagate the loss, we need to differentiate individual spike times t j with respect to their direct input spike times t j−1 and their weights w j . Thankfully, the temporal coding scheme allows us to differentiate Equation ( 1) and obtain the exact derivatives of an output spike time t j with respect to any input t j−1 and weight w j (refer to Comşa et al, 2021 for the full derivation):…”

Section: Learning Spike Times Using Backpropagationmentioning

confidence: 99%

“…Notable existing work in this field includes the SpikeProp model (Bohte et al, 2000) and its extensions (Schrauwen and Van Campenhout, 2004;Booij and tat Nguyen, 2005;McKennoch et al, 2006;Ahmed et al, 2013;Wang et al, 2017;Mostafa, 2018;Hong et al, 2020). In previous work, we have shown that SNNs with temporal coding provide a class of universal approximators for well-behaved functions, and can be trained with backpropagaton to perform classification tasks to accuracies similar to ANNs (Comşa et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Spiking Autoencoders With Temporal Coding

et al. 2021

Self Cite

View full text Add to dashboard Cite

Spiking neural networks with temporal coding schemes process information based on the relative timing of neuronal spikes. In supervised learning tasks, temporal coding allows learning through backpropagation with exact derivatives, and achieves accuracies on par with conventional artificial neural networks. Here we introduce spiking autoencoders with temporal coding and pulses, trained using backpropagation to store and reconstruct images with high fidelity from compact representations. We show that spiking autoencoders with a single layer are able to effectively represent and reconstruct images from the neuromorphically-encoded MNIST and FMNIST datasets. We explore the effect of different spike time target latencies, data noise levels and embedding sizes, as well as the classification performance from the embeddings. The spiking autoencoders achieve results similar to or better than conventional non-spiking autoencoders. We find that inhibition is essential in the functioning of the spiking autoencoders, particularly when the input needs to be memorised for a longer time before the expected output spike times. To reconstruct images with a high target latency, the network learns to accumulate negative evidence and to use the pulses as excitatory triggers for producing the output spikes at the required times. Our results highlight the potential of spiking autoencoders as building blocks for more complex biologically-inspired architectures. We also provide open-source code for the model.

show abstract

Classification of Point Clouds with Neural Networks and Continuum-Type Memories

Reitmann

Kudryashova

Jung

et al. 2021

IFIP Advances in Information and Communication Technology

View full text Add to dashboard Cite

This paper deals with the issue of evaluating and analyzing geometric point sets in three-dimensional space. Point sets or point clouds are often the product of 3D scanners and depth sensors, which are used in the field of autonomous movement for robots and vehicles. Therefore, for the classification of point sets within an active motion, not fully generated point clouds can be used, but knowledge can be extracted from the raw impulses of the respective time points. Attractors consisting of a continuum of stationary states and hysteretic memories can be used to couple multiple inputs over time given non-independent output quantities of a classifier and applied to suitable neural networks. In this paper, we show a way to assign input point clouds to sets of classes using hysteretic memories, which are transferable to neural networks.

show abstract

Temporal Coding in Spiking Neural Networks With Alpha Synaptic Function: Learning With Backpropagation

Cited by 45 publications

References 79 publications

T2FSNN: Deep Spiking Neural Networks with Time-to-first-spike Coding

T2FSNN: Deep Spiking Neural Networks with Time-to-first-spike Coding

Spiking Autoencoders With Temporal Coding

Classification of Point Clouds with Neural Networks and Continuum-Type Memories

Contact Info

Product

Resources

About