Computational neuroscience relies on simulations of neural network models to bridge the gap between the theory of neural networks and the experimentally observed activity dynamics in the brain. The rigorous validation of simulation results against reference data is thus an indispensable part of any simulation workflow. Moreover, the availability of different simulation environments and levels of model description require also validation of model implementations against each other to evaluate their equivalence. Despite rapid advances in the formalized description of models, data, and analysis workflows, there is no accepted consensus regarding the terminology and practical implementation of validation workflows in the context of neural simulations. This situation prevents the generic, unbiased comparison between published models, which is a key element of enhancing reproducibility of computational research in neuroscience. In this study, we argue for the establishment of standardized statistical test metrics that enable the quantitative validation of network models on the level of the population dynamics. Despite the importance of validating the elementary components of a simulation, such as single cell dynamics, building networks from validated building blocks does not entail the validity of the simulation on the network scale. Therefore, we introduce a corresponding set of validation tests and present an example workflow that practically demonstrates the iterative model validation of a spiking neural network model against its reproduction on the SpiNNaker neuromorphic hardware system. We formally implement the workflow using a generic Python library that we introduce for validation tests on neural network activity data. Together with the companion study (Trensch et al., 2018), the work presents a consistent definition, formalization, and implementation of the verification and validation process for neural network simulations.

The reproduction and replication of scientific results is an indispensable aspect of good scientific practice, enabling previous studies to be built upon and increasing our level of confidence in them. However, reproducibility and replicability are not sufficient: an incorrect result will be accurately reproduced if the same incorrect methods are used. For the field of simulations of complex neural networks, the causes of incorrect results vary from insufficient model implementations and data analysis methods, deficiencies in workmanship (e.g., simulation planning, setup, and execution) to errors induced by hardware constraints (e.g., limitations in numerical precision). In order to build credibility, methods such as verification and validation have been developed, but they are not yet well-established in the field of neural network modeling and simulation, partly due to ambiguity concerning the terminology. In this manuscript, we propose a terminology for model verification and validation in the field of neural network modeling and simulation. We outline a rigorous workflow derived from model verification and validation methodologies for increasing model credibility when it is not possible to validate against experimental data. We compare a published minimal spiking network model capable of exhibiting the development of polychronous groups, to its reproduction on the SpiNNaker neuromorphic system, where we consider the dynamics of several selected network states. As a result, by following a formalized process, we show that numerical accuracy is critically important, and even small deviations in the dynamics of individual neurons are expressed in the dynamics at network level.

Recent enhancements in neuroscience, like the development of new and powerful recording techniques of the brain activity combined with the increasing anatomical knowledge provided by atlases and the growing understanding of neuromodulation principles, allow to study the brain at a whole new level, paving the way to the creation of extremely detailed effective network models directly from observed data. Leveraging the advantages of this integrated approach, we propose a method to infer models capable of reproducing the complex spatio-temporal dynamics of the slow waves observed in the experimental recordings of the cortical hemisphere of a mouse under anesthesia. To reliably claim the good match between data and simulations, we implemented a versatile ensemble of analysis tools, applicable to both experimental and simulated data and capable to identify and quantify the spatio-temporal propagation of waves across the cortex. In order to reproduce the observed slow wave dynamics, we introduced an inference procedure composed of two steps: the inner and the outer loop. In the inner loop the parameters of a mean-field model are optimized by likelihood maximization, exploiting the anatomical knowledge to define connectivity priors. The outer loop explores "external" parameters, seeking for an optimal match between the simulation outcome and the data, relying on observables (speed, directions and frequency of the waves) apt for the characterization of cortical slow waves; the outer loop includes a periodic neuro-modulation for a better reproduction of the experimental recordings. We show that our model is capable to reproduce most of the features of the non-stationary and non-linear dynamics displayed by the biological network. Also, the proposed method allows to infer which are the relevant modifications of parameters when the brain state changes, e.g. according to anesthesia levels.

No abstract