Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks

Neftci, Emre; Mostafa, Hesham; Zenke, Friedemann

doi:10.1109/msp.2019.2931595

Cited by 826 publications

(533 citation statements)

References 26 publications

Supporting

Mentioning

530

Contrasting

Unclassified

Order By: Relevance

“…To that end, we computed it using the derivative of the hard threshold nonlinearity of the spikes. As expected, the hard threshold nonlinearity prevented gradient flow into the hidden layer [24], and consequently lead to poor performance (Fig. 2b,c).…”

Section: Resultssupporting

confidence: 75%

“…Second, by constraining their activity through regularization, we showed that surrogate gradients can produce SNNs capable of efficient information processing with sparse spiking activity. Surrogate gradients have been used by a number of studies to train SNNs [24], to solve small-scale toy problems with fractionally predictive neurons [43], to train convolutional SNNs on challenging neuromorphic [44,45] and vision benchmarks [19], or to train recurrent SNNs on temporal problems requiring working memory [21,22]. These studies used a range of different surrogate derivatives ranging from exponential [21], piece-wise linear [22], or tanh [27], sometimes with a non-standard neuron model with a constant leak term [19], but due to the different function choices and datasets they are not easily comparable.…”

Section: Discussionmentioning

confidence: 99%

“…In most studies that rely on surrogate gradients, the surrogate derivative is normalized to 1 [19,21,22,24,25] (cf. Fig.…”

Section: Surrogate Gradient Learning In Recurrent Network Is Sensitimentioning

confidence: 99%

“…The scale of the substituted derivative has a strong effect on the surrogate gradient due to both explicit and implicit forms of recurrence in SNNs ( Fig. 4; [24]). Most notably, the scale may determine whether Surrogate gradient learning is robust to the shape of the surrogate derivative.…”

Section: Surrogate Gradient Learning In Recurrent Network Is Sensitimentioning

confidence: 99%

“…Surrogate gradients have emerged as a solution to build functional SNNs capable of solving complex information processing problems [17][18][19][20][21][22][23][24]. To that end, the actual derivative of a spike, which appears in the analytic expressions of the gradients, is replaced by any well-behaved function.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks

Zenke

Vogels

2020

Preprint

Self Cite

View full text Add to dashboard Cite

AbstractBrains process information in spiking neural networks. Their intricate connections shape the diverse functions these networks perform. In comparison, the functional capabilities of models of spiking networks are still rudimentary. This shortcoming is mainly due to the lack of insight and practical algorithms to construct the necessary connectivity. Any such algorithm typically attempts to build networks by iteratively reducing the error compared to a desired output. But assigning credit to hidden units in multi-layered spiking networks has remained challenging due to the non-differentiable nonlinearity of spikes. To avoid this issue, one can employ surrogate gradients to discover the required connectivity in spiking network models. However, the choice of a surrogate is not unique, raising the question of how its implementation influences the effectiveness of the method. Here, we use numerical simulations to systematically study how essential design parameters of surrogate gradients impact learning performance on a range of classification problems. We show that surrogate gradient learning is robust to different shapes of underlying surrogate derivatives, but the choice of the derivative’s scale can substantially affect learning performance. When we combine surrogate gradients with a suitable activity regularization technique, robust information processing can be achieved in spiking networks even at the sparse activity limit. Our study provides a systematic account of the remarkable robustness of surrogate gradient learning and serves as a practical guide to model functional spiking neural networks.

show abstract

Section: Resultssupporting

confidence: 75%

Section: Discussionmentioning

confidence: 99%

“…In most studies that rely on surrogate gradients, the surrogate derivative is normalized to 1 [19,21,22,24,25] (cf. Fig.…”

Section: Surrogate Gradient Learning In Recurrent Network Is Sensitimentioning

confidence: 99%

Section: Surrogate Gradient Learning In Recurrent Network Is Sensitimentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks

Zenke

Vogels

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Overview of Recent Advancements in Deep Learning and Artificial Intelligence

Narayanan,

Cao,

Panda

et al. 2023

Advances in Electromagnetics Empowered by Artificial Intelligence and Deep Learning

View full text Add to dashboard Cite

Complex Oxides for Brain‐Inspired Computing: A Review

et al. 2022

View full text Add to dashboard Cite

The fields of brain‐inspired computing, robotics, and, more broadly, artificial intelligence (AI) seek to implement knowledge gleaned from the natural world into human‐designed electronics and machines. In this review, the opportunities presented by complex oxides, a class of electronic ceramic materials whose properties can be elegantly tuned by doping, electron interactions, and a variety of external stimuli near room temperature, are discussed. The review begins with a discussion of natural intelligence at the elementary level in the nervous system, followed by collective intelligence and learning at the animal colony level mediated by social interactions. An important aspect highlighted is the vast spatial and temporal scales involved in learning and memory. The focus then turns to collective phenomena, such as metal‐to‐insulator transitions (MITs), ferroelectricity, and related examples, to highlight recent demonstrations of artificial neurons, synapses, and circuits and their learning. First‐principles theoretical treatments of the electronic structure, and in situ synchrotron spectroscopy of operating devices are then discussed. The implementation of the experimental characteristics into neural networks and algorithm design is then revewed. Finally, outstanding materials challenges that require a microscopic understanding of the physical mechanisms, which will be essential for advancing the frontiers of neuromorphic computing, are highlighted.

show abstract

Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks

Cited by 826 publications

References 26 publications

The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks

The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks

Overview of Recent Advancements in Deep Learning and Artificial Intelligence

Complex Oxides for Brain‐Inspired Computing: A Review

Contact Info

Product

Resources

About