Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks

Fang, Wei; Yu, Zhaofei; Chen, Yanqi; Masquelier, Timothée; Huang, Tiejun; Tian, Yonghong

doi:10.1109/iccv48922.2021.00266

Cited by 331 publications

(289 citation statements)

References 48 publications

Supporting

Mentioning

283

Contrasting

Unclassified

Order By: Relevance

“…The classification accuracy of the proposed network along with those of other CSNNs trained by different learning strategies including surrogate gradient learning, ANN-to-SNN conversion, and tandem learning are presented in Table 3. Our proposed network could reach 93.11% categorization accuracy on Cifar10 with T = 60 and outperform any other CSNN trained listed in Table 3, except Fang et al (2021) [31] that use surrogate gradient in CSNNs with Leaky-IF neurons with trainable membrane time constants (i.e., each spiking neuron layer has an independent and trainable membrane time constant). Although, they reached 0.04% better accuracy than us, implementing large CSNNs with Leaky-IF neurons having different time constants, independent of the implementation platform, is highly expensive in terms of memory and computation.…”

Section: Cifar10mentioning

confidence: 84%

“…Interestingly, our network with proxy learning could surpass other networks trained with surrogate gradient learning (SGL). To do a fair comparison, we trained the same CSNN as ours using surrogate gradient learning method (with arctangent surrogate function [31]) that reached to the best accuracy of 94.41% with T = 50. We also trained a CANN with the same architecture to our CSNN using backpropagation that reached to 94.60% accuracy best (it is 0.04% better than proxy learning).…”

Section: Fashion-mnistmentioning

confidence: 99%

See 1 more Smart Citation

Spiking neural networks trained via proxy

Kheradpisheh,

Mirsadeghi,

Masquelier

2021

Preprint

View full text Add to dashboard Cite

We propose a new learning algorithm to train spiking neural networks (SNN) using conventional artificial neural networks (ANN) as proxy. We couple two SNN and ANN networks, respectively, made of integrate-and-fire (IF) and ReLU neurons with the same network architectures and shared synaptic weights. The forward passes of the two networks are totally independent. By assuming IF neuron with rate-coding as an approximation of ReLU, we backpropagate the error of the SNN in the proxy ANN to update the shared weights, simply by replacing the ANN final output with that of the SNN. We applied the proposed proxy learning to deep convolutional SNNs and evaluated it on two benchmarked datasets of Fahion-MNIST and Cifar10 with 94.56% and 93.11% classification accuracy, respectively. The proposed networks could outperform other deep SNNs trained with tandem learning, surrogate gradient learning, or converted from deep ANNs. Converted SNNs require long simulation times to reach reasonable accuracies while our proxy learning leads to efficient SNNs with much shorter simulation times.

show abstract

Section: Cifar10mentioning

confidence: 84%

Section: Fashion-mnistmentioning

confidence: 99%

Spiking neural networks trained via proxy

Kheradpisheh,

Mirsadeghi,

Masquelier

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…While this is a simple method to reduce SNN latency, it may potentially have limitations in neuromorphic chip designs in terms of spike routing or parallel spike processing capability. It is worth mentioning here that additional optimizations like learnable membrane time constants (Rathi and Roy, 2020 ; Fang et al, 2021b ), network architectures like Residual networks (Fang et al, 2021a ), conversion error calibration techniques (Deng and Gu, 2021 ; Li et al, 2021 ), hybrid spike encoding (Datta et al, 2021 ) are complementary to the current proposal and can be augmented in the algorithm to further minimize the inference latency. Tables 2 , 3 therefore includes primarily basic SNN architectures based on IF nodes without any additional optimizations to substantiate the importance and interpretability of the need for layerwise threshold optimization.…”

Section: Experiments and Resultsmentioning

confidence: 99%

Neuroevolution Guided Hybrid Spiking Neural Network Training

Sengupta

2022

Front. Neurosci.

View full text Add to dashboard Cite

Neuromorphic computing algorithms based on Spiking Neural Networks (SNNs) are evolving to be a disruptive technology driving machine learning research. The overarching goal of this work is to develop a structured algorithmic framework for SNN training that optimizes unique SNN-specific properties like neuron spiking threshold using neuroevolution as a feedback strategy. We provide extensive results for this hybrid bio-inspired training strategy and show that such a feedback-based learning approach leads to explainable neuromorphic systems that adapt to the specific underlying application. Our analysis reveals 53.8, 28.8, and 28.2% latency improvement for the neuroevolution-based SNN training strategy on CIFAR-10, CIFAR-100, and ImageNet datasets, respectively in contrast to state-of-the-art conversion based approaches. The proposed algorithm can be easily extended to other application domains like image classification in presence of adversarial attacks where 43.2 and 27.9% latency improvements were observed on CIFAR-10 and CIFAR-100 datasets, respectively.

show abstract

“…Acc. GCN [28] 54.00 LIF-SCNN [5] 60.50 tdBN-RseNet18 [6] 67.80 PLIF [29](T=20) 74.80 LIF-R18 -From scratch 53.64 LIF-R18pipe-D (reuse ImageNet Exp.) 62.89 LIF-R18pipe-D (reuse ES-ImageNet Exp.)…”

Section: Network Pretrain Methodsmentioning

confidence: 99%

Rethinking Pretraining as a Bridge from ANNs to SNNs

Lin¹,

Hu²,

Ma³

et al. 2022

Preprint

View full text Add to dashboard Cite

Spiking neural networks (SNNs) are known as a typical kind of brain-inspired models with their unique features of rich neuronal dynamics, diverse coding schemes and low power consumption properties. How to obtain a high-accuracy model has always been the main challenge in the field of SNN. Currently, there are two mainstream methods, i.e., obtaining a converted SNN through converting a well-trained Artificial Neural Network (ANN) to its SNN counterpart or training an SNN directly. However, the inference time of a converted SNN is too long, while SNN training is generally very costly and inefficient. In this work, a new SNN training paradigm is proposed by combining the concepts of the two different training methods with the help of the pretrain technique and BP-based deep SNN training mechanism. We believe that the proposed paradigm is a more efficient pipeline for training SNNs. The pipeline includes pipe-S for static data transfer tasks and pipe-D for dynamic data transfer tasks. SOTA results are obtained in a large-scale event-driven dataset ES-ImageNet. For training acceleration, we achieve the same (or higher) best accuracy as similar LIF-SNNs using 1/10 training time on ImageNet-1K and 2/5 training time on ES-ImageNet and also provide a time-accuracy benchmark for a new dataset ES-UCF101. These experimental results reveal the similarity of the functions of parameters between ANNs and SNNs and also demonstrate the various potential applications of this SNN training pipeline.

show abstract

Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks

Cited by 331 publications

References 48 publications

Spiking neural networks trained via proxy

Spiking neural networks trained via proxy

Neuroevolution Guided Hybrid Spiking Neural Network Training

Rethinking Pretraining as a Bridge from ANNs to SNNs

Contact Info

Product

Resources

About