A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks

Wu, Jibin; Chua, Yansong; Zhang, Malu; Li, Guoqi; Li, Haizhou; Tan, Kay Chen

doi:10.1109/tnnls.2021.3095724

Cited by 100 publications

(62 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interestingly, our proposed CSNN with proxy learning could significantly outperform CSNNs with tandem learning rule [27,28]. This might be due to inconsistency between the forward and backward passes of tandem learning.…”

Section: Cifar10mentioning

confidence: 88%

“…To reach the optimal classification accuracy, they usually require at least several hundred of inference time steps. Tandem learning methods [27,28] could resolve these issues in conversion methods. Same as our proxy learn-ing, they assume that IF neurons with rate-code approximate artificial neurons with ReLU activation.…”

Section: Discussionmentioning

confidence: 99%

“…In short, the IF neuron remains silent if the output of ReLU is zero, and it will fire with shorter delay for larger ReLU outputs. For rate coding, the higher ReLU outputs correspond to higher firing rates in IF neuron [27,28,29].…”

Section: If Approximating Relumentioning

confidence: 99%

“…The forth approach is the tandem learning which is consists of an SNN and an ANN coupled layerwise through weight sharing [27,28]. The auxiliary ANN is used to facilitate the error backpropagation for the training of the SNN at the spike-train level, while the SNN is used to derive the exact spiking neural representation.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Spiking neural networks trained via proxy

Kheradpisheh,

Mirsadeghi,

Masquelier

2021

Preprint

View full text Add to dashboard Cite

We propose a new learning algorithm to train spiking neural networks (SNN) using conventional artificial neural networks (ANN) as proxy. We couple two SNN and ANN networks, respectively, made of integrate-and-fire (IF) and ReLU neurons with the same network architectures and shared synaptic weights. The forward passes of the two networks are totally independent. By assuming IF neuron with rate-coding as an approximation of ReLU, we backpropagate the error of the SNN in the proxy ANN to update the shared weights, simply by replacing the ANN final output with that of the SNN. We applied the proposed proxy learning to deep convolutional SNNs and evaluated it on two benchmarked datasets of Fahion-MNIST and Cifar10 with 94.56% and 93.11% classification accuracy, respectively. The proposed networks could outperform other deep SNNs trained with tandem learning, surrogate gradient learning, or converted from deep ANNs. Converted SNNs require long simulation times to reach reasonable accuracies while our proxy learning leads to efficient SNNs with much shorter simulation times.

show abstract

Section: Cifar10mentioning

confidence: 88%

Section: Discussionmentioning

confidence: 99%

Section: If Approximating Relumentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Spiking neural networks trained via proxy

Kheradpisheh,

Mirsadeghi,

Masquelier

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…• Backpropagation: this class of approaches try to overcome the non-differentiable nature of SNNs in order to enable spike-based backpropagation. Most approaches (a) try to approximate the derivatives to make gradient learning possible [25], (b) employ differentiable surrogate activations to replace the non-differentiable threshold operation [31] or (c) use a tandem approach where a SNN is performing the prediction and an equivalent standard NN is adapting the weights using backpropagation [32]. • Conversion: here the idea is to circumvent the backpropagation problem by training a standard ANN and reusing the trained weights for the SNN.…”

Section: Spiking Neural Network Designmentioning

confidence: 99%

Efficient Hardware Acceleration of Sparsely Active Convolutional Spiking Neural Networks

Sommer¹,

Özkan²,

Keszöcze³

et al. 2022

Preprint

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) compute in an event-based matter to achieve a more efficient computation than standard Neural Networks. In SNNs, neuronal outputs (i.e. activations) are not encoded with real-valued activations but with sequences of binary spikes. The motivation of using SNNs over conventional neural networks is rooted in the special computational aspects of spike based processing, especially the very high degree of sparsity of neural output activations. Well established architectures for conventional Convolutional Neural Networks (CNNs) feature large spatial arrays of Processing Elements (PEs) that remain highly underutilized in the face of activation sparsity. We propose a novel architecture that is optimized for the processing of Convolutional SNNs (CSNNs) that feature a high degree of activation sparsity. In our architecture, the main strategy is to use less but highly utilized PEs. The PE array used to perform the convolution is only as large as the kernel size, allowing all PEs to be active as long as there are spikes to process. This constant flow of spikes is ensured by compressing the feature maps (i.e. the activations) into queues that can then be processed spike by spike. This compression is performed in runtime using dedicated circuitry, leading to a self-timed scheduling. This allows the processing time to scale directly with the number of spikes. A novel memory organization scheme called memory interlacing is used to efficiently store and retrieve the membrane potentials of the individual neurons using multiple small parallel on-chip RAMs. Each RAM is hardwired to its PE, reducing switching circuitry and allowing RAMs to be located in close proximity to the respective PE. We implemented the proposed architecture on an FPGA and achieved a significant speedup compared to other implementations while needing less hardware resources and maintaining a lower energy consumption.

show abstract