High-Performance Computing Framework Based on Distributed Systems for Large-Scale Neurophysiological Data

Abstract

Due to the significantly drastic progress and improvement in neurophysiological recording technologies, neuroscientists have faced various complexities dealing with unstructured large-scale neural data. In the neuroscience community, these complexities could create serious bottlenecks in storing, sharing, and processing neural datasets. In this article, we developed a distributed high-performance computing (HPC) framework called `Big neuronal data framework' (BNDF), to overcome these complexities. BNDF is based on open-source big data frameworks, Hadoop and Spark providing a flexible and scalable structure. We examined BNDF on three different large-scale electrophysiological recording datasets from nonhuman primate’s brains. Our results exhibited faster runtimes with scalability due to the distributed nature of BNDF. We compared BNDF results to a widely used platform like MATLAB in an equitable computational resource. Compared with other similar methods, using BNDF provides more than five times faster performance in spike sorting as a usual neuroscience application.

From End to End: Gaining, Sorting, and Employing High-Density Neural Single Unit Recordings

“…If we record with 1,000 channels at a sampling frequency of 40 kHz, an hour is just enough to produce 30 GB of electrophysiological data (Hadianpour et al, 2021 ); furthermore, a Neuropixels probe of 384 channels generates 90 GBs when sampling is set to 30 kHz. Unless virtually infinite storage capacity is in our hands, compression is what we should make use of.…”

“…When searching for unified computational for a more specific use, P-sort is a unique pipeline and software, destined to sort cerebellar single unit activities (Sedaghat-Nejad et al, 2021 ). CellExplorer suits for single-neuron characterization and signal visualization are for those who are confident with MATLAB (Petersen et al, 2020 ), whereas Big Neuronal Data Format (BNDF) emphasizes large-scale data processing and reducing overall runtime (Hadianpour et al, 2021 ). Combinato, on the other hand, is written in C/C++ and enables dealing with long-term, noisy recordings mostly in an unsupervised fashion (Knieling et al, 2016 ).…”