Exploring Complex Brain-Simulation Workloads on Multi-GPU Deployments

Abstract

In-silico brain simulations are the de-facto tools computational neuroscientists use to understand large-scale and complex brain-function dynamics. Current brain simulators do not scale efficiently enough to largescale problem sizes (e.g., >100,000 neurons) when simulating biophysically complex neuron models. The goal of this work is to explore the use of true multi-GPU acceleration through NVIDIA's GPUDirect technology on computationally challenging brain models and to assess their scalability. The brain model used is a state-of-the-art, extended Hodgkin-Huxley, biophysically meaningful, three-compartmental model of the inferior-olivary nucleus. The Hodgkin-Huxley model is the most widely adopted conductance-based neuron representation, and thus the results from simulating this representative workload are relevant for many other brain experiments. Not only the actual network-simulation times but also the network-setup times were taken into account when designing and benchmarking the multi-GPU version, an aspect often ignored in similar previous work. Network sizes varying from 65K to 2M cells, with 10 and 1,000 synapses per neuron were executed on 8, 16, 24, and 32 GPUs. Without loss of generality, simulations were run for 100 ms of biological time. Findings indicate that communication overheads do not dominate overall execution while scaling the network size up is computationally tractable. This scalable design proves that large-network simulations of complex neural models are possible using a multi-GPU design with GPUDirect.

ExaFlexHH: an exascale-ready, flexible multi-FPGA library for biologically plausible brain simulations

“…Notably, ExaFlexHH maintains a consistent performance trend, particularly when the DFEs are fully utilized with data. In contrast, the work in Vlag et al (2019) reported a variable speedup, dropping to as low as 8× under similar simulations. Furthermore, Chatzikonstantis et al (2019) in fact reported a decrease in performance during scaling out in their experiments for the case of uniform connectivity distributions and a connectivity density of 1,000 synapses per neuron, which is much lower than that supported by ExaFlexHH.…”

“…This is not typical of multi-node setups. For illustrative purposes, we compared the performance scalability of our implementation against two other works simulating an IO network: a multi-GPU setup supporting GPUDirect, as detailed in Vlag et al (2019), and a multi-node many-core CPU architecture as described in Chatzikonstantis et al (2019). The results presented in Table 4 demonstrate superior performance scalability for ExaFlexHH.…”

EDEN: A High-Performance, General-Purpose, NeuroML-Based Neural Simulator

“…By using this scheme, information is transmitted efficiently in large clusters: no information has to be exchanged between nodes that do not communicate with each other. This is a scalability improvement over existing methods, where the full matrix of connectivity degrees among nodes is gathered on all nodes (Vlag et al, 2019 ; Magalhães and Schürmann, 2020 ).…”

Efficient parameter calibration and real-time simulation of large-scale spiking neural networks with GeNN and NEST

“…A number of GPU-based simulators are currently in use for SNN simulation, such as ANNarchy (Vitay et al, 2015), CARLsim (Niedermeier et al, 2022), BINDSnet (Hazan et al, 2018), GeNN, and NEST GPU (Golosio et al, 2021). These use different design principles (Brette and Goodman, 2012;Vlag et al, 2019) that are optimal for specific use cases. NEST GPU, for example, follows the design principle of NEST allowing for the distributed simulation of very large networks on multiple GPUs (on multiple machines) using MPI.…”