Larger GPU-accelerated brain simulations with procedural connectivity

Abstract

Large-scale simulations of spiking neural network models are an important tool for improving our understanding of the dynamics and ultimately the function of brains. However, even small mammals such as mice have on the order of 1 × 10 12 synaptic connections which, in simulations, are each typically charaterized by at least one floatingpoint value. This amounts to several terabytes of data -an unrealistic memory requirement for a single desktop machine. Large models are therefore typically simulated on distributed supercomputers which is costly and limits large-scale modelling to a few privileged research groups. In this work, we describe extensions to GeNN -our Graphical Processing Unit (GPU) accelerated spiking neural network simulator -that enable it to 'procedurally' generate connectivity and synaptic weights 'on the go' as spikes are triggered, instead of storing and retrieving them from memory. We find that GPUs are wellsuited to this approach because of their raw computational power which, due to memory bandwidth limitations, is often under-utilised when simulating spiking neural networks. We demonstrate the value of our approach with a recent model of the Macaque visual cortex consisting of 4.13 × 10 6 neurons and 24.2 × 10 9 synapses. Using our new method, it can be simulated on a single GPU -a significant step forward in making large-scale brain modelling accessible to many more researchers. Our results match those obtained on a supercomputer and the simulation runs up to 35 % faster on a single high-end GPU than previously on over 1000 supercomputer nodes.spiking neural networks | GPU | high-performance computing | brain simulation T he brain of a mouse has around 70 × 10 6 neurons, but 1 this number is dwarfed by the 1 × 10 12 synapses which 2 connect them (1). In computer simulations of spiking neu-3 ral networks, propagating spikes involves adding the synaptic 4 input from each spiking presynaptic neuron to the postsy-5 naptic neurons. The information describing which neurons 6 are synaptically connected and with what weight is typically 7 generated before a simulation is run and stored in large arrays. 8For large-scale brain models this creates high memory require-9 ments, so that they can typically only be simulated on large 10 distributed computer systems using software such as NEST (2) 11 or NEURON (3). By careful design, these simulators can keep 12 the memory requirements for each node constant, even when a 13 simulation is distributed across thousands of nodes (4). How-14 ever, high performance computer (HPC) systems are bulky, 15 expensive and consume a lot of power and are hence typi-16 cally shared resources, only accessible to a limited number of 17 researchers and for time-limited investigations. 18Neuromorphic systems (5-9) take inspiration from the brain 19 and have been developed specifically for simulating large spik-20 ing neural networks more efficiently. One particular relevant 21 feature of the brain is that its memory elements -the synapses 22 Unfortunately, propagating a s...

Usage and Scaling of an Open-Source Spiking Multi-Area Model of Monkey Cortex

“…The multi-area model of monkey cortex developed by Schmidt et al [11,12] and described here has a somewhat higher threshold for reuse, due to its greater complexity and specificity. Nevertheless, it has already been ported to a single GPU using connectivity generated on the fly each time a spike is triggered, thereby trading memory storage and retrieval for computation, which is possible in this case because the synapses are static [52]. We hope that the technologies presented here push the complexity barrier of neuroscience modeling a bit further out, such that the model will find a wide uptake and serve as a scaffold for generating an ever more complete and realistic picture of cortical structure, dynamics, and function.…”

Scalability of large neural network simulations via activity tracking with time asynchrony and procedural connectivity

“…Because the models are very close dynamically and structurally, it is a good way to evaluate roughly the memory and computational performances of corresponding simulators and hardware solutions. Indeed, simulations in [26, 46, 24] use LIF models that are dynamically very close to the Hawkes model used here (see Section 2.1). Structurally, we compare networks of similar size and connectivity.…”

“…Table 1) and both computational and memory complexities of their solutions. However, in [26, 46, 24] we were not able to find the mean firing network rate and there is no computational complexity except in [24], where only the memory complexity is provided. Still these simulations are meant to be biologically plausible and therefore should have been run on realistic parameter ranges.…”

“…In comparison to [26], by combining procedural connectivity with activity tracking and time asynchrony, the new algorithm of the present work, called ATiTAP, obtains a smaller memory cost and a range of computational costs, which is much smaller than the sum of the costs of 4600 cores, and this, by using a single processing unit (PU) (either processor or core) instead of using several parallel PUs.…”

Simulation of Large Scale Neural Models With Event-Driven Connectivity Generation

“…Originally, the idea of event-driven connectivity generation has been proposed in the case of abstract neurons for which spike timing is exactly known, i.e., rule-based artificial cell units, or finite state machines (Lytton and Stewart, 2006 ). This approach has then been applied with integrate-and-fire (IF) neurons, i.e., quadratic IF (Izhikevich and Edelman, 2008 ) and leaky IF over GPU hardware (Knight and Nowotny, 2020 ).…”