Backpropagation With Sparsity Regularization for Spiking Neural Network Learning

Abstract

The spiking neural network (SNN) is a possible pathway for low-power and energy-efficient processing and computing exploiting spiking-driven and sparsity features of biological systems. This article proposes a sparsity-driven SNN learning algorithm, namely backpropagation with sparsity regularization (BPSR), aiming to achieve improved spiking and synaptic sparsity. Backpropagation incorporating spiking regularization is utilized to minimize the spiking firing rate with guaranteed accuracy. Backpropagation realizes the temporal information capture and extends to the spiking recurrent layer to support brain-like structure learning. The rewiring mechanism with synaptic regularization is suggested to further mitigate the redundancy of the network structure. Rewiring based on weight and gradient regulates the pruning and growth of synapses. Experimental results demonstrate that the network learned by BPSR has synaptic sparsity and is highly similar to the biological system. It not only balances the accuracy and firing rate, but also facilitates SNN learning by suppressing the information redundancy. We evaluate the proposed BPSR on the visual dataset MNIST, N-MNIST, and CIFAR10, and further test it on the sensor dataset MIT-BIH and gas sensor. Results bespeak that our algorithm achieves comparable or superior accuracy compared to related works, with sparse spikes and synapses.

Sparse-Firing Regularization Methods for Spiking Neural Networks with Time-to-First-Spike Coding

“…Previous studies have developed methods that suppress the firing of SNNs in the framework of the surrogate gradient method [26,27]. They applied direct regularization to the spike variable s(t) ∈ {0, 1} represented at each time step in the model to the time-discretized SNN.…”

“…( 2), which integrates the membrane potential. By contrast, M-SSR, unlike the previous method [26,27], can be transformed from the time-integration form to the timing form by setting v → V th . This may correspond to the fact that the learning method with the surrogate function can transition to a timing-like learning method by taking a limit [19].…”

“…Recently, it has become possible to train relatively large models [25]. However, with few exceptions [26,27], the neurons exhibit high firing frequencies, and it is debatable whether the timing information is efficiently utilized.…”

Neuromorphic Computing for Interactive Robotics: A Systematic Review

“…Here the magnitude of synaptic plasticity is assumed by the rate of pre-and post-synaptic firing over the specific time period. This type of model is typically used for converting conventional artificial neural networks to spiking neural networks by using backpropagation [55] [56]. The ANN is trained with backpropagation technique and then it is converted into an equivalent SNN by relating the activation of ANN units and the firing rate of spiking neurons.…”

Energy Efficient Training of SNN using Local Zeroth Order Method

“…(Alawad et al, 2017) uses stochastic neurons to increase energy efficiency during inference. More recently (Yan et al, 2022) uses regularization during the training in order to increase the sparsity of spikes which reduces computational burden and energy consumption. (Cramer et al, 2022) performs the forward pass on a neuromorphic chip, while the backward pass still takes place on a standard GPU.…”