Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks

Abstract

Spiking Neural Networks (SNNs) are promising energyefficient models for neuromorphic computing. For training the non-differentiable SNN models, the backpropagation through time (BPTT) with surrogate gradients (SG) method has achieved high performance. However, this method suffers from considerable memory cost and training time during training. In this paper, we propose the Spatial Learning Through Time (SLTT) method that can achieve high performance while greatly improving training efficiency compared with BPTT. First, we show that the backpropagation of SNNs through the temporal domain contributes just a little to the final calculated gradients. Thus, we propose to ignore the unimportant routes in the computational graph during backpropagation. The proposed method reduces the number of scalar multiplications and achieves a small memory occupation that is independent of the total time steps. Furthermore, we propose a variant of SLTT, called SLTT-K, that allows backpropagation only at K time steps, then the required number of scalar multiplications is further reduced and is independent of the total time steps. Experiments on both static and neuromorphic datasets demonstrate superior training efficiency and performance of our SLTT. In particular, our method achieves state-of-the-art accuracy on ImageNet, while the memory cost and training time are reduced by more than 70% and 50%, respectively, compared with BPTT.

Shrinking Your TimeStep: Towards Low-Latency Neuromorphic Object Recognition with Spiking Neural Networks

“…Even though Spikformer (Zhou et al 2023) uses a more complex transformer structure and data augmentation, the accuracy of our SSNN is still 5.08% higher than that of Spikformer. When the average timestep is increased to 8, SSNN achieves an accuracy of 78.57%, surpassing SLTT (Meng et al 2023)…”

Sharing leaky-integrate-and-fire neurons for memory-efficient spiking neural networks

“…This observation has been presented in the previous work Chowdhury et al (2021b); Li et al (2023b,a) where they show SNN can work with very low timestep 1 ∼ 2. Note that the approaches to memory reduction proposed by other works, such as those reducing simulation time step Chowdhury et al (2021b) and reducing SNN time dependence Meng et al (2023), can be combined with our layer/channel-wise sharing technique. This would lead to an even more significant decrease in memory usage, demonstrating the compatibility and potential of our method when integrated with other optimization strategies.…”