Neural Architecture Search for Spiking Neural Networks

Kim, Youngeun; Li, Yuhang; Park, Hyoungseob; Venkatesha, Yeshwanth; Panda, Priyadarshini

doi:10.48550/arxiv.2201.10355

Cited by 8 publications

(8 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…TET [50] studies the choice of the loss function to provide better convergence in SNNs. [51] employs neural architecture search tailored for SNN.…”

Section: Direct Training Snnmentioning

confidence: 99%

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

Li¹,

Deng²,

Dong³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Spiking Neural Network (SNN), originating from the neural behavior in biology, has been recognized as one of the nextgeneration neural networks. Conventionally, SNNs can be obtained by converting from pre-trained Artificial Neural Networks (ANNs) by replacing the non-linear activation with spiking neurons without changing the parameters. In this work, we argue that simply copying and pasting the weights of ANN to SNN inevitably results in activation mismatch, especially for ANNs that are trained with batch normalization (BN) layers. To tackle the activation mismatch issue, we first provide a theoretical analysis by decomposing local conversion error to clipping error and flooring error, and then quantitatively measure how this error propagates throughout the layers using the second-order analysis. Motivated by the theoretical results, we propose a set of layer-wise parameter calibration algorithms, which adjusts the parameters to minimize the activation mismatch. Extensive experiments for the proposed algorithms are performed on modern architectures and large-scale tasks including ImageNet classification and MS COCO detection. We demonstrate that our method can handle the SNN conversion with batch normalization layers and effectively preserve the high accuracy even in 32 time steps. For example, our calibration algorithms can increase up to 65% accuracy when converting VGG-16 with BN layers.

show abstract

“…TET [50] studies the choice of the loss function to provide better convergence in SNNs. [51] employs neural architecture search tailored for SNN.…”

Section: Direct Training Snnmentioning

confidence: 99%

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

Li¹,

Deng²,

Dong³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We note that the model agnostic property of MAML renders it complementary to techniques that improve the performance of SNNs, such as batch normalization [50] and dropout [14], or methods to train SNNs such as parameter conversions [51,52] and neural architecture search [53] such as those applied to SNNs [54]. The latter methods can be directly integrated an additional step in the outer loop update.…”

Section: Discussionmentioning

confidence: 99%

Meta-learning spiking neural networks with surrogate gradient descent

Stewart

Neftci

2022

Neuromorph. Comput. Eng.

View full text Add to dashboard Cite

Adaptive "life-long'' learning at the edge and during online task performance is an aspirational goal of AI research. Neuromorphic hardware implementing SNN are particularly attractive in this regard, as their real-time, event-based, local computing paradigm makes them suitable for edge implementations and fast learning. However, the long and iterative learning that characterizes state-of-the-art SNN training is incompatible with the physical nature and real-time operation of neuromorphic hardware. Bi-level learning, such as meta-learning is increasingly used in deep learning to overcome these limitations. In this work, we demonstrate gradient-based meta-learning in SNN using the surrogate gradient method that approximates the spiking threshold function for gradient estimations. Because surrogate gradients can be made twice differentiable, well-established, and effective second-order gradient meta-learning methods such as MAML can be used. We show that SNN meta-trained using MAML perform comparably to conventional ANN meta-trained with MAML on event-based meta-datasets. Furthermore, we demonstrate the specific advantages that accrue from meta-learning: fast learning without the requirement of high precision weights or gradients, training-to-learn with quantization and mitigating the effects of approximate synaptic plasticity rules. Our results emphasize how meta-learning techniques can become instrumental for deploying neuromorphic learning technologies on real-world problems.

show abstract

“…SNNs Architecture : While the majority of existing works on SNNs have focused on the image classification problem and utilize available ANN architectures such as VGG or Resnet, having an appropriate SNN architecture is critical. Recently, meta-learning such as neural architecture search (NAS) has been utilized to find the best SNN architecture [ 93 ].…”

Section: Conclusion and Future Perspectivesmentioning

confidence: 99%

Spiking Neural Networks and Their Applications: A Review

et al. 2022

View full text Add to dashboard Cite

The past decade has witnessed the great success of deep neural networks in various domains. However, deep neural networks are very resource-intensive in terms of energy consumption, data requirements, and high computational costs. With the recent increasing need for the autonomy of machines in the real world, e.g., self-driving vehicles, drones, and collaborative robots, exploitation of deep neural networks in those applications has been actively investigated. In those applications, energy and computational efficiencies are especially important because of the need for real-time responses and the limited energy supply. A promising solution to these previously infeasible applications has recently been given by biologically plausible spiking neural networks. Spiking neural networks aim to bridge the gap between neuroscience and machine learning, using biologically realistic models of neurons to carry out the computation. Due to their functional similarity to the biological neural network, spiking neural networks can embrace the sparsity found in biology and are highly compatible with temporal code. Our contributions in this work are: (i) we give a comprehensive review of theories of biological neurons; (ii) we present various existing spike-based neuron models, which have been studied in neuroscience; (iii) we detail synapse models; (iv) we provide a review of artificial neural networks; (v) we provide detailed guidance on how to train spike-based neuron models; (vi) we revise available spike-based neuron frameworks that have been developed to support implementing spiking neural networks; (vii) finally, we cover existing spiking neural network applications in computer vision and robotics domains. The paper concludes with discussions of future perspectives.

show abstract

Neural Architecture Search for Spiking Neural Networks

Cited by 8 publications

References 51 publications

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration

Meta-learning spiking neural networks with surrogate gradient descent

Spiking Neural Networks and Their Applications: A Review

Contact Info

Product

Resources

About