Spike: A GPU Optimised Spiking Neural Network Simulator

Abstract

Spiking Neural Network (SNN) simulations require internal variables -such as the membrane voltages of individual neurons and their synaptic inputs -to be updated on a sub-millisecond resolution. As a result, a single second of simulation time requires many thousands of update calculations per neuron. Furthermore, increases in the scale of SNN models have, accordingly, led to manyfold increases in the runtime of SNN simulations. Existing solutions to this problem of scale include high performance CPU based simulators capable of multithreaded execution ("CPU parallelism"). More recent GPU based simulators have emerged, which aim to utilise GPU parallelism for SNN execution. We have identified several key speedups, which give GPU based simulators up to an order of magnitude performance increase over CPU based simulators on several benchmarks. We present the Spike simulator with three key optimisations: timestep grouping, active synapse grouping, and delay insensitivity. Combined, these optimisations massively increase the speed of executing a SNN simulation and produce a simulator which is, on a single machine, faster than currently available simulators.

Even Faster SNN Simulation with Lazy+Event-driven Plasticity and Shared Atomics

“…The versatility, and specifically the ability and ease of defining neuron models can serve as another way to categorize simulators. General purpose simulators allow the user to freely define the behavior of neurons and synapses [1], [5], [7], [31]- [35], whereas the rest only allow the use of a fixed number of predefined models [6], [18], [26], [30].…”

“…Another important classification can be made based on the approach adopted regarding the high-level software implementation. In one category, the time-drive approach simulates the behavior of neurons and synapses in lock-step, advancing all of them in every iteration of the simulation [1], [6], [7], [23], [33], [36], [37]. In contrast, event-driven simulators only update a neuron or synapse when a new event affects the (predictable, up until that time instant) evolution of their state [1], [6], [23], [33], [36], [37].…”

“…In one category, the time-drive approach simulates the behavior of neurons and synapses in lock-step, advancing all of them in every iteration of the simulation [1], [6], [7], [23], [33], [36], [37]. In contrast, event-driven simulators only update a neuron or synapse when a new event affects the (predictable, up until that time instant) evolution of their state [1], [6], [23], [33], [36], [37]. Ease of implementation and per-element efficiency are two strong points of the time-driven approaches, however if the time-step is set very low (desired to increase simulation accuracy), the simulation becomes computationally inefficient.…”

“…As a middle ground, approaches that adopt characteristics of both categories can be called hybrid [38], [39]. It is debatable whether approaches that adopt a time-driven neuron update with an event-driven synapse update (event-driven plasticity) [2]- [6] should be called hybrid, since this combination is less integral.…”

“…In this paper we present two optimizations targeting steps (2) and (3) of the SNN simulation loop. In order to accelerate step (2) we, for the first time, combine lazy plasticity [1] with event-driven plasticity [2]- [6] and efficiently facilitate the computation of pre-and post-synaptic spikes via per-neuron bit fields and integer intrinsics. Combined, these measures yield a cumulative speedup of 30× over the naïve approach of blindly updating each synapse at every loop iteration and are 1.5×-2× faster than our closest competitor.…”