Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Deng, Shikuang; Li, Yuhang; Zhang, Shanghang; Gu, Shi

doi:10.48550/arxiv.2202.11946

Cited by 21 publications

(41 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…TdBN, BNTT [48], [49] improve direct training by adapting batch normalization layers in SNNs. TET [50] studies the choice of the loss function to provide better convergence in SNNs. [51] employs neural architecture search tailored for SNN.…”

Section: Direct Training Snnmentioning

confidence: 99%

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

Li¹,

Deng²,

Dong³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Spiking Neural Network (SNN), originating from the neural behavior in biology, has been recognized as one of the nextgeneration neural networks. Conventionally, SNNs can be obtained by converting from pre-trained Artificial Neural Networks (ANNs) by replacing the non-linear activation with spiking neurons without changing the parameters. In this work, we argue that simply copying and pasting the weights of ANN to SNN inevitably results in activation mismatch, especially for ANNs that are trained with batch normalization (BN) layers. To tackle the activation mismatch issue, we first provide a theoretical analysis by decomposing local conversion error to clipping error and flooring error, and then quantitatively measure how this error propagates throughout the layers using the second-order analysis. Motivated by the theoretical results, we propose a set of layer-wise parameter calibration algorithms, which adjusts the parameters to minimize the activation mismatch. Extensive experiments for the proposed algorithms are performed on modern architectures and large-scale tasks including ImageNet classification and MS COCO detection. We demonstrate that our method can handle the SNN conversion with batch normalization layers and effectively preserve the high accuracy even in 32 time steps. For example, our calibration algorithms can increase up to 65% accuracy when converting VGG-16 with BN layers.

show abstract

Section: Direct Training Snnmentioning

confidence: 99%

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

Li¹,

Deng²,

Dong³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Specifically, Swin Transformer ( 17) is used as the baseline and the ANN-to-SNN conversion algorithm is implemented as the training method. Next, we validate the effectiveness of our model and compare our model with other state-of-the-art methods, including RMP (18), RNL (19), QCFS (20), SNNC-AP (21), Hybrid (22), tdBN (23), TET (24) and DSR (5), for image classification tasks on Cifar-10 (25), Cifar-100 (26), and ImageNet datasets (27). Further experiments have been conducted to check the performance of our model with ultralow time steps.…”

Section: Resultsmentioning

confidence: 99%

Efficient Spiking Transformer Enabled By Partial Information

Wang¹,

Fang²,

Cao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Spiking neural networks (SNNs) have received substantial attention in recent years due to their sparse and asynchronous communication nature, and thus can be deployed in neuromorphic hardware and achieve extremely high energy efficiency. However, SNNs currently can hardly realize a comparable performance to that of artificial neural networks (ANNs) because their limited scalability does not allow for large-scale networks. Especially for Transformer, as a model of ANNs that has accomplished remarkable performance in various machine learning tasks, its implementation in SNNs by conventional methods requires a large number of neurons, notably in the self-attention module. Inspired by the mechanisms in the nervous system, we propose an efficient spiking Transformer (EST) framework enabled by partial information to address the above problem. In this model, we not only implemented the self-attention module with a reasonable number of neurons, but also introduced partial-information self-attention (PSA), which utilizes only partial input signals, further reducing computational resources compared to conventional methods. The experimental results show that our EST can outperform the state-of-the-art SNN model in terms of accuracy and the number of time steps on both Cifar-10/100 and ImageNet datasets. In particular, the proposed EST model achieves 78.48% top-1 accuracy on the ImageNet dataset with only 16 time steps. In addition, our proposed PSA reduces flops by 49.8% with negligible performance loss compared to a self-attention module with full information.

show abstract

“…) as suggested by . Now, denote the overall spiking neural network as a function f T (x), its forward propagation can be formulated as (Deng et al, 2022) or by conversion (Bu et al, 2022b).…”

Section: Spiking Neural Networkmentioning

confidence: 99%

Neuromorphic Data Augmentation for Training Spiking Neural Networks

Kim

Park

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) have recently become more popular as a biologically plausible substitute for traditional Artificial Neural Networks (ANNs). SNNs are cost-efficient and deployment-friendly because they process input in both spatial and temporal manners using binary spikes. However, we observe that the information capacity in SNNs is affected by the number of timesteps, leading to an accuracy-efficiency tradeoff. In this work, we study a fine-grained adjustment of the number of timesteps in SNNs. Specifically, we treat the number of timesteps as a variable conditioned on different input samples to reduce redundant timesteps for certain data. We call our method Spiking Early-Exit Neural Networks (SEENNs). To determine the appropriate number of timesteps, we propose SEENN-I which uses a confidence score thresholding to filter out the uncertain predictions, and SEENN-II which determines the number of timesteps by reinforcement learning. Moreover, we demonstrate that SEENN is compatible with both the directly trained SNN and the ANN-SNN conversion. By dynamically adjusting the number of timesteps, our SEENN achieves a remarkable reduction in the average number of timesteps during inference. For example, our SEENN-II ResNet-19 can achieve 96.1% accuracy with an average of 1.08 timesteps on the CIFAR-10 test dataset.

show abstract

Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting

Cited by 21 publications

References 14 publications

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

Efficient Spiking Transformer Enabled By Partial Information

Neuromorphic Data Augmentation for Training Spiking Neural Networks

Contact Info

Product

Resources

About