The Backpropagation Algorithm Implemented on Spiking Neuromorphic Hardware

Renner, Alpha; Sheldon, Forrest; Zlotnik, Anatoly; Tao, Louis; Sornborger, Andrew

doi:10.48550/arxiv.2106.07030

Cited by 6 publications

(7 citation statements)

References 77 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The neural dynamics for all neurons are modelled using the LIF model, which is a common representation of biological spiking neuron dynamics [8], [26], [37]. The differential equation describing the dynamics (the membrane voltage) of a LIF neuron is:…”

Section: Methodology: Preliminariesmentioning

confidence: 99%

See 1 more Smart Citation

Spiking Neural Networks for Visual Place Recognition via Weighted Neuronal Assignments

Hussaini,

Milford,

Fischer

2021

Preprint

View full text Add to dashboard Cite

Spiking neural networks (SNNs) offer both compelling potential advantages, including energy efficiency and low latencies, and challenges including the non-differentiable nature of event spikes. Much of the initial research in this area has converted deep neural networks to equivalent SNNs, but this conversion approach potentially negates some of the potential advantages of SNN-based approaches developed from scratch. One promising area for high performance SNNs is template matching and image recognition. This research introduces the first high performance SNN for the Visual Place Recognition (VPR) task: given a query image, the SNN has to find the closest match out of a list of reference images. At the core of this new system is a novel assignment scheme that implements a form of ambiguity-informed salience, by up-weighting single-placeencoding neurons and down-weighting "ambiguous" neurons that respond to multiple different reference places. In a range of experiments on the challenging Oxford RobotCar and Nordland datasets, we show that our SNN achieves comparable VPR performance to state-of-the-art and classical techniques, and degrades gracefully in performance with an increasing number of reference places. Our results provide a significant milestone towards SNNs that can provide robust, energy-efficient and low latency robot localization.

show abstract

Section: Methodology: Preliminariesmentioning

confidence: 99%

“…Due to the non-differentiable neuronal dynamics of spiking neurons, supervised techniques such as back-propagation cannot be applied in a straightforward manner. Recently, Renner et al [26] and Lee at al. [27] proposed algorithms that approximate backpropagation, and demonstrated use on Intel's Loihi neuromorphic hardware.…”

Section: A Spiking Neural Networkmentioning

confidence: 98%

Spiking Neural Networks for Visual Place Recognition via Weighted Neuronal Assignments

Hussaini,

Milford,

Fischer

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In general, this motivates offline training of SNNs typically using GPUs, where deployment can take place on low-power SNN accelerators. Several recent studies have leveraged programmable microcode of neuromorphic research processors to adopt BPTT variants on a single chip [48,49] Training SNNs has traditionally been slow compared to ANNs. This is due to the additional sequential operations needed in recurrent networks, along with stateful computations implemented at the neuron node.…”

Section: Low-cost Inferencementioning

confidence: 99%

Exploiting deep learning accelerators for neuromorphic workloads

Sun,

Titterton,

Gopiani

et al. 2024

Neuromorph. Comput. Eng.

View full text Add to dashboard Cite

Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency when performing inference with deep learning workloads. Error backpropagation is presently regarded as the most effective method for training SNNs, but in a twist of irony, when training on modern graphics processing units (GPUs) this becomes more expensive than non-spiking networks. The emergence of Graphcore's Intelligence Processing Units (IPUs) balances the parallelized nature of deep learning workloads with the sequential, reusable, and sparsified nature of operations prevalent when training SNNs. IPUs adopt multi-instruction multi-data (MIMD) parallelism by running individual processing threads on smaller data blocks, which is a natural fit for the sequential, non-vectorized steps required to solve spiking neuron dynamical state equations. We present an IPU-optimized release of our custom SNN Python package, snnTorch, which exploits fine-grained parallelism by utilizing low-level, pre-compiled custom operations to accelerate irregular and sparse data access patterns that are characteristic of training SNN workloads. We provide a rigorous performance assessment across a suite of commonly used spiking neuron models, and propose methods to further reduce training run-time via half-precision training. By amortizing the cost of sequential processing into vectorizable population codes, we ultimately demonstrate the potential for integrating domain-specific accelerators with the next generation of neural networks.

show abstract

“…By potentiating GATING neurons, spikes emanating from the clock selectively enable information propagation only when and where needed. In this way, information is dynamically guided through the circuit with the appropriate processing occurring only at the appropriate times and locations for the computation to proceed (see [34] for another example of this sort of gating). We note that our implementation here assumes the regularity of our gating clock to organize timing (as may be done with the Loihi chip), however, even in a noisy circuit with spike jitter, this regular relative timing can be arranged with synfire-gated synfire chains made of up neuronal populations.…”

Section: Two's Complement Multiplicationmentioning

confidence: 99%

Binary operations on neuromorphic hardware with application to linear algebraic operations and stochastic equations

Iaroshenko

Sornborger

Arana

2023

Neuromorph. Comput. Eng.

View full text Add to dashboard Cite

Non-von Neumann computational hardware, based on neuron-inspired, non-linear elements connected via linear, weighted synapses -- so-called neuromorphic systems -- is a viable computational substrate. Since neuromorphic systems have been shown to use less power than CPUs for many applications, they are of potential use in autonomous systems such as robots, drones, and satellites, for which power resources are at a premium. The power used by neuromorphic systems is approximately proportional to the number of spiking events produced by neurons on-chip. However, typical information encoding on these chips is in the form of firing rates that unarily encode information. That is, the number of spikes generated by a neuron is meant to be proportional to an encoded value used in a computation or algorithm. Unary encoding is less efficient (produces more spikes) than binary encoding. For this reason, here we present neuromorphic computational mechanisms for implementing binary two's complement operations. We use the mechanisms to construct a neuromorphic, binary matrix multiplication algorithm that may be used as a primitive for linear differential equation integration, deep networks, and other standard calculations. We also construct a random walk circuit and apply it in Brownian motion simulations. We study how both algorithms scale in circuit size and iteration time.

show abstract

The Backpropagation Algorithm Implemented on Spiking Neuromorphic Hardware

Cited by 6 publications

References 77 publications

Spiking Neural Networks for Visual Place Recognition via Weighted Neuronal Assignments

Spiking Neural Networks for Visual Place Recognition via Weighted Neuronal Assignments

Exploiting deep learning accelerators for neuromorphic workloads

Binary operations on neuromorphic hardware with application to linear algebraic operations and stochastic equations

Contact Info

Product

Resources

About