A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration

Li, Yuhang; Deng, Shikuang; Dong, Xin; Gong, Ruihao; Gu, Shi

doi:10.48550/arxiv.2106.06984

Cited by 11 publications

(22 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Spike-timing-dependent plasticity (STDP) reinforces or punishes the neuronal connection based on the spike history [5,6,36,41,90,97]. Also, a line of work [20,30,31,52,72,73,98] approximate ReLU with LIF by converting pre-trained ANNs to SNNs using weight or threshold balancing.…”

Section: Spiking Neural Networkmentioning

confidence: 99%

See 1 more Smart Citation

Neural Architecture Search for Spiking Neural Networks

Kim¹,

Li²,

Park³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) have gained huge attention as a potential energy-efficient alternative to conventional Artificial Neural Networks (ANNs) due to their inherent high-sparsity activation. However, most prior SNN methods use ANN-like architectures (e.g., VGG-Net or ResNet), which could provide sub-optimal performance for temporal sequence processing of binary information in SNNs. To address this, in this paper, we introduce a novel Neural Architecture Search (NAS) approach for finding better SNN architectures. Inspired by recent NAS approaches that find the optimal architecture from activation patterns at initialization, we select the architecture that can represent diverse spike activation patterns across different data samples without training. Furthermore, to leverage the temporal correlation among the spikes, we search for feed forward connections as well as backward connections (i.e., temporal feedback connections) between layers. Interestingly, SNASNet found by our search algorithm achieves higher performance with backward connections, demonstrating the importance of designing SNN architecture for suitably using temporal information. We conduct extensive experiments on three image recognition benchmarks where we show that SNASNet achieves state-of-the-art performance with significantly lower timesteps (5 timesteps). The code has been released at https://github.com/Intelligent-Computing-Lab-Yale/Neural-Architecture-Search-for-Spiking-Neural-Networks.

show abstract

Section: Spiking Neural Networkmentioning

confidence: 99%

“…Finally, we compare the memory and computational efficiency between SNASNet and previous works [31,52,68] in Table 5. In the table, we also compare SNASNet-Fw-APx and SNASNet-Bw-APx where x is the kernel size of AvgPooling layer for the vectorize block (we use x=2 in our default setting).…”

Section: Memory and Computational Efficiencymentioning

confidence: 99%

Neural Architecture Search for Spiking Neural Networks

Kim¹,

Li²,

Park³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, the key disadvantage of DNN-to-SNN conversion is that it yields SNNs with much higher latency compared to other techniques. Some previous research [16], [24] proposed to down-scale the threshold term to train low-latency SNNs, but the scaling factor was either a hyperparameter or obtained via linear grid-search, and the latency needed for convergence still remained large (>64).…”

Section: B Dnn-to-snn Conversionmentioning

confidence: 99%

“…To further reduce the conversion error, [15] minimized the difference between the DNN and SNN post-activation values for each layer. To do this, the activation function of the IF SNN must first be derived [15], [16]. We assume that the initial membrane potential of a layer l (U l (0)) is 0.…”

Section: B Dnn-to-snn Conversionmentioning

confidence: 99%

“…• We analytically and empirically show that the primary source of error in current DNN-to-SNN conversion strategies [15], [16] is the incorrect and simplistic model of the distributions of DNN and SNN activations. • We propose a novel DNN-to-SNN conversion and finetuning algorithm that reduces the conversion error for ultra low latencies by accurately capturing these distributions and thus minimizing the difference between SNN and DNN activation functions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Can Deep Neural Networks be Converted to Ultra Low-Latency Spiking Neural Networks?

Datta¹,

A.²

2021

Preprint

View full text Add to dashboard Cite

Spiking neural networks (SNNs), that operate via binary spikes distributed over time, have emerged as a promising energy efficient ML paradigm for resource-constrained devices. However, the current state-of-the-art (SOTA) SNNs require multiple time steps for acceptable inference accuracy, increasing spiking activity and, consequently, energy consumption. SOTA training strategies for SNNs involve conversion from a non-spiking deep neural network (DNN). In this paper, we determine that SOTA conversion strategies cannot yield ultra low latency because they incorrectly assume that the DNN and SNN pre-activation values are uniformly distributed. We propose a new training algorithm that accurately captures these distributions, minimizing the error between the DNN and converted SNN. The resulting SNNs have ultra low latency and high activation sparsity, yielding significant improvements in compute efficiency. In particular, we evaluate our framework on image recognition tasks from CIFAR-10 and CIFAR-100 datasets on several VGG and ResNet architectures. We obtain top-1 accuracy of 64.19% with only 2 time steps on the CIFAR-100 dataset with ∼159.2× lower compute energy compared to an iso-architecture standard DNN. Compared to other SOTA SNN models, our models perform inference 2.5-8× faster (i.e., with fewer time steps).

show abstract

Cerebron: A Reconfigurable Architecture for Spatiotemporal Sparse Spiking Neural Networks

Chen

Gao

2022

IEEE Trans. VLSI Syst.

View full text Add to dashboard Cite

Spiking neural networks (SNNs) are promising 1 alternatives to artificial neural networks (ANNs) since they 2 are more realistic brain-inspired computing models. SNNs have 3 sparse neuron firing over time, i.e., spatiotemporal sparsity; thus, 4 they are helpful in enabling energy-efficient hardware inference. 5 However, exploiting the spatiotemporal sparsity of SNNs in 6 hardware leads to unpredictable and unbalanced workloads, 7 degrading the energy efficiency. Compared to SNNs with sim-8 ple fully connected structures, those extensive structures (e.g., 9 standard convolutions, depthwise convolutions, and pointwise 10 convolutions) can deal with more complicated tasks but lead 11 to difficulties in hardware mapping. In this work, we propose 12 a novel reconfigurable architecture, Cerebron, which can fully 13 exploit the spatiotemporal sparsity in SNNs with maximized 14 data reuse and propose optimization techniques to improve the 15 efficiency and flexibility of the hardware. To achieve flexibility, 16 the reconfigurable compute engine is compatible with a variety of 17 spiking layers and supports inter-computing-unit (CU) and intra-18 CU reconfiguration. The compute engine can exploit data reuse 19 and guarantee parallel data access when processing different 20 convolutions to achieve memory efficiency. A two-step data 21 sparsity exploitation method is introduced to leverage the sparsity 22 of discrete spikes and reduce the computation time. Besides, 23 an online channelwise workload scheduling strategy is designed 24 to reduce the latency further. Cerebron is verified on image 25 segmentation and classification tasks using a variety of state-of-26 the-art spiking network structures. Experimental results show 27 that Cerebron has achieved at least 17.5× prediction energy 28 reduction and 20× speedup compared with state-of-the-art field-29 programmable gate array (FPGA)-based accelerators.30

show abstract

A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibration

Cited by 11 publications

References 28 publications

Neural Architecture Search for Spiking Neural Networks

Neural Architecture Search for Spiking Neural Networks

Can Deep Neural Networks be Converted to Ultra Low-Latency Spiking Neural Networks?

Cerebron: A Reconfigurable Architecture for Spatiotemporal Sparse Spiking Neural Networks

Contact Info

Product

Resources

About