Reproducibility in systems biology modelling

Abstract

The reproducibility crisis has emerged as an important concern across many fields of science including life science, since many published results failed to reproduce. Systems biology modelling, which involves mathematical representation of biological processes to study complex system behaviour, was expected to be least affected by this crisis. While lack of reproducibility of experimental results and computational analysis could be a repercussion of several compounded factors, it was not fully understood why systems biology models with well defined mathematical expressions fail to reproduce and how prevalent it is. Hence, we systematically attempted to reproduce 455 kinetic models of biological processes published in peer-reviewed research articles from 152 journals; which is collectively a work of about 1400 scientists from 49 countries. Our investigation revealed that about half (49%) of the models could not be reproduced using the information provided in the published manuscripts. With further effort, an additional 12% of the models could be reproduced either by empirical correction or support from authors. The other 37% remained non-reproducible models due to missing parameter values, missing initial concentration, inconsistent model structure, or a combination of these factors. Among the corresponding authors of the non-reproducible model we contacted, less than 30% responded. Our analysis revealed that models published in journals across several fields of life science failed to reproduce, revealing a common problem in the peer-review process. Hence, we propose an 8-point reproducibility scorecard that can be used by authors, reviewers and journal editors to assess each model and address the reproducibility crisis.

Characteristics of mathematical modeling languages that facilitate model reuse in systems biology: A software engineering perspective

“…The first step in building a model consisting of several submodels is to regenerate the individual parts from the literature. This can already be a challenge due to several issues with reproducibility in systems biology including incomplete model descriptions, errors in formulas, availability of the code or missing descriptions of experiment setup or design choices [6][7][8]. A recent study by curators of the BioModels database showed that only 51% of 455 published models were directly reproducible [8].…”

“…This can already be a challenge due to several issues with reproducibility in systems biology including incomplete model descriptions, errors in formulas, availability of the code or missing descriptions of experiment setup or design choices [6][7][8]. A recent study by curators of the BioModels database showed that only 51% of 455 published models were directly reproducible [8]. In an extreme case, Topalidou et al [9] report requiring three months to reproduce a neuroscientific model of the basal ganglia.…”

“…For example, the BioModels database [27] features curated models in SBML format, but most of these models are monolithic and therefore require further modification if only parts of the model should be reused [28]. In the aforementioned reproducibility study, the curators of this database found that the reproducibility rate for SBML models was only slightly higher (56%) than the overall rate of reproducibility across all models (51%; including models written in SBML, MATLAB, Python, C, R, and other languages) [8]. Our previous example of the translation of the SHM also shows that using a suitable language is a necessary but not sufficient criterion for the model to actually be reusable.…”

Countering reproducibility issues in mathematical models with software engineering techniques: A case study using a one-dimensional mathematical model of the atrioventricular node

“…We believe that results from this case study will be applicable to a large set of systems biology models for several reasons: The Inada model exhibits all of the common reproducibility issues identified by the BioModels reproducibility study [4]-"recoverable" issues like sign errors, missing equations, order of magnitude, and unit errors, as well as "non-recoverable" issues such as missing parameter values,…”

“…According to Stodden et al [3], only an estimated 26% of 204 randomly selected computational articles in the journal Science were reproducible-despite of the journal's policy that requires researchers to provide their data upon request. Curators of the BioModels database recently found that of 455 ordinary differential equation (ODE) models, only 51% were directly reproducible and even after contacting the authors and manual corrections by experts, 37% remained non-reproducible [4]. As a single extreme case, Topalidou et al [5] reported requiring three months to reproduce a neuroscientific model of the basal ganglia.…”