Reproducibility of computational workflows is automated using continuous analysis

Abstract

Replication, validation and extension of experiments are crucial for scientific progress. Computational experiments are inherently scriptable and should be easy to reproduce. However, it remains difficult and time consuming to reproduce computational results because analyses are designed and run in a specific computing environment, which may be difficult or impossible to match from written instructions. We report a workflow named continuous analysis that can build reproducibility into computational analyses. Continuous analysis combines Docker, a container technology akin to virtual machines, with continuous integration, a software development technique, to automatically re-run a computational analysis whenever updates or improvements are made to source code or data. This allows results to be accurately reproduced without needing to contact the study authors. Continuous analysis allows reviewers, editors or readers to verify reproducibility without manually downloading and re-running any code and can provide an audit trail for analyses of data that cannot be shared.

FMRIPrep: a robust preprocessing pipeline for functional MRI

“…Community-centered, peer-reviewed development, and adoption of fMRIPrep describes how the community interacts, discusses the code review process, and underscores how the modular design of fMRIPrep successfully facilitates contributions from peers. Finally, fMRIPrep undergoes continuous integration testing (see Online Methods, Figure S5), a technique that has recently been proposed as a means to ensure reproducibility of analyses in computational sciences 45,46 .…”

Practical Computational Reproducibility in the Life Sciences

“…While Conda and Bioconda provide an excellent solution for packaging software components and their dependencies, archiving them, and recreating analysis environments, they are still dependent on and can be influenced by the host computer system (Beaulieu-Jones and Greene 2017). Moreover, since Conda packages are frequently updated, if a Conda virtual environment is created by specifying only the top-level tools and versions, recreating it at a later point in time using the same specifications may easily result in slightly different dependencies being installed.…”

Developing a modern data workflow for evolving data

“…While designed as a software development tool, continuous integration has features which are useful for automating the management of evolving data: it detects changes in files, automates running code, and tests output for consistency. Because these tasks are also useful in a research context, this lead to the suggestion that continuous analysis could be used to drive research pipelines (Beaulieu-Jones and Greene, 2017). We expand on this concept by applying continuous integration to the management of evolving data.…”