Novel Knowledge Distillation to Improve Training Accuracy of Spin-based SNN

Li, Hanrui; Lone, Aijaz H.; Tian, Fengshi; Yang, Jie; El‐Atab, Nazek

doi:10.1109/aicas57966.2023.10168575

Cited by 1 publication

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…IDSNN outperformed all other methods in terms of accuracy using the same or smaller time-step (

). Compared to three of the latest and best-performing distillation-related works from 2023, under the condition of using time-step of four (i.e., less than the three other works), our method achieved 0.62%, 2.81%, and 8.6% higher accuracy than KDSNN [ 17 ], LaSNN [ 16 ], and [ 26 ], respectively. In particular, when using ANNs that had the same accuracy (95.10%) as teachers, the accuracy of IDSNN was 0.62 higher than KDSNN.…”

Section: Resultsmentioning

confidence: 87%

“…It could achieve an accuracy of 75.24 using a time-step of four and 75.41 using a time-step of six. Compared with the KD training method [ 26 ], IDSNN also achieved better performance (75.24 vs. 74.42) using a shorter time-step (four vs. five). Compared to direct SNN training method Dspike [ 27 ] and Real Spike [ 29 ], IDSNN surpassed them by 1.89 and 10.37, respectively, in terms of accuracy when the time-step was four.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

IDSNN: Towards High-Performance and Low-Latency SNN Training via Initialization and Distillation

Fan,

Zhang,

Zhang

2023

Biomimetics

View full text Add to dashboard Cite

Spiking neural networks (SNNs) are widely recognized for their biomimetic and efficient computing features. They utilize spikes to encode and transmit information. Despite the many advantages of SNNs, they suffer from the problems of low accuracy and large inference latency, which are, respectively, caused by the direct training and conversion from artificial neural network (ANN) training methods. Aiming to address these limitations, we propose a novel training pipeline (called IDSNN) based on parameter initialization and knowledge distillation, using ANN as a parameter source and teacher. IDSNN maximizes the knowledge extracted from ANNs and achieves competitive top-1 accuracy for CIFAR10 (94.22%) and CIFAR100 (75.41%) with low latency. More importantly, it can achieve 14× faster convergence speed than directly training SNNs under limited training resources, which demonstrates its practical value in applications.

show abstract

“…IDSNN outperformed all other methods in terms of accuracy using the same or smaller time-step (

Section: Resultsmentioning

confidence: 87%

Section: Resultsmentioning

confidence: 99%