Sparse Spiking Gradient Descent

Abstract

There is an increasing interest in emulating Spiking Neural Networks (SNNs) on neuromorphic computing devices due to their low energy consumption. Recent advances have allowed training SNNs to a point where they start to compete with traditional Artificial Neural Networks (ANNs) in terms of accuracy, while at the same time being energy efficient when run on neuromorphic hardware. However, the process of training SNNs is still based on dense tensor operations originally developed for ANNs which do not leverage the spatiotemporally sparse nature of SNNs. We present here the first sparse SNN backpropagation algorithm which achieves the same or better accuracy as current state of the art methods while being significantly faster and more memory efficient. We show the effectiveness of our method on real datasets of varying complexity (Fashion-MNIST, Neuromophic-MNIST and Spiking Heidelberg Digits) achieving a speedup in the backward pass of up to 70x, and 40% more memory efficient, without losing accuracy.Preprint. Under review.

The fine line between dead neurons and sparsity in binarized spiking neural networks

“…Kheradpisheh et al use temporal backpropagation (BS4NN) on dense BSNNs (Kheradpisheh et al, 2021). We further consider several additional promising lightweight approaches to training full precision SNNs, including sparse spiking gradient descent (SSGD) which reduces overhead during gradient calculations (Perez-Nieves & Goodman, 2021), and neural heterogeneity (NH) which compresses the required number of neurons with the addition of neuronindependent parameters (Perez-Nieves et al, 2021). Interestingly, BSNNs with threshold annealing can outperform full precision SNNs in terms of accuracy using considerably less memory for model parameters.…”

Training Spiking Neural Networks Using Lessons From Deep Learning

“…Recent experiments have shown that rate-coded networks (at the output) are robust to sparsity-promoting regularization terms [110], [111], [113]. However, networks that rely on time-to-first-spike schemes have had less success, which is unsurprising given that temporal outputs are already sparse.…”

EGRU: Event-based GRU for activity-sparse inference and learning

“…Using pseudo-derivatives for backpropagating through the non-differential threshold function, as we use for our discrete-time EGRU, was originally proposed for feedforward spiking networks in neuromorphic hardware in [16] and developed further in [3,66]. The sparsity of learning with BPTT when using appropriate pseudo-derivatives in a discrete-time feed-forward spiking neural network was recently described in [50].…”