Challenges in Reproducibility, Replicability, and Comparability of Computational Models and Tools for Neuronal and Glial Networks, Cells, and Subcellular Structures

Abstract

The possibility to replicate and reproduce published research results is one of the biggest challenges in all areas of science. In computational neuroscience, there are thousands of models available. However, it is rarely possible to reimplement the models based on the information in the original publication, let alone rerun the models just because the model implementations have not been made publicly available. We evaluate and discuss the comparability of a versatile choice of simulation tools: tools for biochemical reactions and spiking neuronal networks, and relatively new tools for growth in cell cultures. The replicability and reproducibility issues are considered for computational models that are equally diverse, including the models for intracellular signal transduction of neurons and glial cells, in addition to single glial cells, neuron-glia interactions, and selected examples of spiking neuronal networks. We also address the comparability of the simulation results with one another to comprehend if the studied models can be used to answer similar research questions. In addition to presenting the challenges in reproducibility and replicability of published results in computational neuroscience, we highlight the need for developing recommendations and good practices for publishing simulation tools and computational models. Model validation and flexible model description must be an integral part of the tool used to simulate and develop computational models. Constant improvement on experimental techniques and recording protocols leads to increasing knowledge about the biophysical mechanisms in neural systems. This poses new challenges for computational neuroscience: extended or completely new computational methods and models may be required. Careful evaluation and categorization of the existing models and tools provide a foundation for these future needs, for constructing multiscale models or extending the models to incorporate additional or more detailed biophysical mechanisms. Improving the quality of publications in computational neuroscience, enabling progressive building of advanced computational models and tools, can be achieved only through adopting publishing standards which underline replicability and reproducibility of research results.

Brian 2: an intuitive and efficient neural simulator

“…Finally, a major issue in computational science generally, and computational neuroscience in particular, is the crisis of reproducibility of computational models (LeVeque et al, 2012;Eglen et al, 2017;Podlaski et al, 2017;Manninen et al, 2018).…”

Neuroinformatics and Computational Modelling as Complementary Tools for Neurotoxicology Studies

“…Such models can be used to shed light on the cellular and molecular mechanisms that contribute to the initiation and maintenance of, for example, short‐ and long‐term plasticity, learning and even the consolidation of memories . There exist also models that address gap junctions in glial networks and other phenomena common to non‐neuronal cells, such as astrocytes .…”

“…Some existing models cannot be effortlessly reproduced based on the information presented in scientific publications, making comparisons impossible (see, e.g. . To ensure the testability and comparability of in silico models, it is thus essential that all in silico models and their source codes are stored in databases.…”

“…To ensure the testability and comparability of in silico models, it is thus essential that all in silico models and their source codes are stored in databases. Currently, the source codes written in different programming languages are only limitedly available in model repositories . Therefore, the description of in silico models, their source codes and experimental data used to construct and validate the in silico models should be openly available to the scientific community .…”

Code Generation in Computational Neuroscience: A Review of Tools and Techniques

“…Publications in computational neuroscience mostly contain descriptions of models in terms of their mathematical equations and the algorithms to add additional behavior such as the mechanism for threshold detection and spike generation. However, if looking at a model implementation and comparing it to the corresponding published model description, one often finds that they are not in agreement due to the complexity and variety of available forms of abstractions of such a transformation (e.g., Manninen et al, 2017 , 2018 ). Using a general purpose programming language to express the model implementation even aggravates this problem as such languages provide full freedom for model developers while lacking the means to guide them in their challenging task due to the absence of neuroscience domain concepts.…”