Training Spiking Neural Networks with Accumulated Spiking Flow

Abstract

The fast development of neuromorphic hardwares promotes Spiking Neural Networks (SNNs) to a thrilling research avenue. Current SNNs, though much efficient, are less effective compared with leading Artificial Neural Networks (ANNs) especially in supervised learning tasks. Recent efforts further demonstrate the potential of SNNs in supervised learning by introducing approximated backpropagation (BP) methods. To deal with the non-differentiable spike function in SNNs, these BP methods utilize information from the spatio-temporal domain to adjust the model parameters. With the increasing of time window and network size, the computational complexity of spatio-temporal backpropagation augments dramatically. In this paper, we propose a new backpropagation method for SNNs based on the accumulated spiking flow (ASF), i.e. ASF-BP. In the proposed ASF-BP method, updating parameters does not rely on the spike train of spiking neurons but leverage accumulated inputs and outputs of spiking neurons over the time window, which reduces the BP complexity significantly. We further present an adaptive linear estimation model to approach the dynamic characteristics of spiking neurons statistically. Experimental results demonstrate that with our proposed ASF-BP method, light-weight convolutional SNNs achieve superior performances compared with other spike-based BP methods on both non-neuromorphic (MNIST, CIFAR10) and neuromorphic (CIFAR10-DVS) datasets. The code is available at https://github.com/neural-lab/ASF-BP.

Hebbian semi-supervised learning in a sample efficiency setting

“…Networks trained with this approach reach accuracy comparable with traditional DNN models, but suitable for implementation in energy-efficient neuromorphic computing. Results close to the latter are achieved in other works in which similar approaches are taken [169,217,218,220,221,227], where similar rate coding schemes are used for the output, but the spiking differentiation problem is solved by using smoothing kernels for the spikes. Specific approaches address the problem of achieving fast inference, in addition to accurate predictions, which can be obtained by leveraging TTFS coding, as in [159,228].…”

Rethinking skip connections in Spiking Neural Networks with Time-To-First-Spike coding

“…While these techniques are generally effective, they require a multitude of timesteps to emulate float activations using binary spikes. A set of recent studies have suggested the use of surrogate functions to circumvent this non-differentiable backpropagation issue (Lee et al, 2016 , 2020 ; Shrestha and Orchard, 2018 ; Wu et al, 2018 , 2020 , 2021 ; Neftci et al, 2019 ; Li et al, 2021b ; Kim et al, 2022a ). These methods, accounting for temporal dynamics during weight training, exhibit high performance and short latency.…”

Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks

“…Furthermore, the conversion method is not suitable for neuromorphic data. Some gradient-based direct training methods find the equivalence between spike representations (e.g., firing rates or first spike times) of SNNs and some differentiable mappings or fixedpoint equations [40,43,58,61,62,66,67,69,75]. Then the spike-representation-based methods train SNNs by gradients calculated from the corresponding mappings or fixedpoint equations.…”