A surrogate gradient spiking baseline for speech command recognition

Abstract

Artificial neural networks (ANNs) are the basis of recent advances in artificial intelligence (AI); they typically use real valued neuron responses. By contrast, biological neurons are known to operate using spike trains. In principle, spiking neural networks (SNNs) may have a greater representational capability than ANNs, especially for time series such as speech; however their adoption has been held back by both a lack of stable training algorithms and a lack of compatible baselines. We begin with a fairly thorough review of literature around the conjunction of ANNs and SNNs. Focusing on surrogate gradient approaches, we proceed to define a simple but relevant evaluation based on recent speech command tasks. After evaluating a representative selection of architectures, we show that a combination of adaptation, recurrence and surrogate gradients can yield light spiking architectures that are not only able to compete with ANN solutions, but also retain a high degree of compatibility with them in modern deep learning frameworks. We conclude tangibly that SNNs are appropriate for future research in AI, in particular for speech processing applications, and more speculatively that they may also assist in inference about biological function.

Co-learning synaptic delays, weights and adaptation in spiking neural networks

“…In this study, we use the adaptive leaky integrate-and-fire neuron (AdLIF) model (Bittar and Garner, 2022) with updated parameter boundaries. To highlight this modification, our neuron model is called the constrained AdLIF, (cAdLIF).…”

“…This contrasts with the adaptive neuronal threshold adaptation, in which only the spiking activity is taken into account (Yin et al, 2021). This model was chosen because of its proven superior performance in comparison with a leaky integrate-and-fire (LIF) model with an adaptive neuronal threshold (Bittar and Garner, 2022). The computational graph of the neuron model, rolled out over time, is shown in Figure 2.…”

“…In these methods, surrogate gradients are used to approximate the non-differentiability, which was introduced by the thresholding mechanism (Heaviside function), δs [t] δu [t] , with a differentiable function in the backward pass of error-backpropagation. For simplicity and comparability with the previous study (Bittar and Garner, 2022), the boxcar surrogate gradient function, shown in Equation ( 6), was used in this study.…”

“…These parameters were initialized following a uniform distribution: α ∈ [0.36, 0.96], β ∈ [0.96, 0.99], a ∈ [0, 1], and b ∈ [0, 2]. Different from the AdLIF model (Bittar and Garner, 2022), we extended the available range of the membrane potential decay parameter α and limited the dependency of the adaptation current with respect to the membrane potential, a, to remain positive, which showed to stabilize the neuron model for sparse input data. During the training process, all neuron parameters are clipped to remain within these boundaries.…”

SoftHebb: Bayesian inference in unsupervised Hebbian soft winner-take-all networks

“…Namely, neuromorphic computing has been advancing machine intelligence in energy efficiency, and recent evidence shows that it improves also conventional metrics of SOTA performance such as accuracy, reward, or speed. For example, spike-based models achieve processing speed and energy efficiency through the imitation of biological neuronal activations, without trading off performance (Jeffares et al 2022), or with minimal trade-offs (Bittar and Garner 2022); short-term plasticity (STP) improves the performance of neural networks in dynamic tasks such as video processing, navigation, robotics, and video games (Moraitis et al 2020, Garcia Rodriguez et al c) Classification: SoftHebb's unsupervised algorithm can be used to cluster an input dataset into classes, either on its own through its Bayesian inference of the classes as hidden causes of the input, or with an added supervised linear classifier. E.g.…”