CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility

Abstract

The traditional scientific paper falls short of effectively communicating computational research. To help improve this situation, we propose a system by which the computational workflows underlying research articles are checked. The CODECHECK system uses open infrastructure and tools and can be integrated into review and publication processes in multiple ways. We describe these integrations along multiple dimensions (importance, who, openness, when). In collaboration with academic publishers and conferences, we demonstrate CODECHECK with 25 reproductions of diverse scientific publications. These CODECHECKs show that asking for reproducible workflows during a collaborative review can effectively improve executability. While CODECHECK has clear limitations, it may represent a building block in Open Science and publishing ecosystems for improving the reproducibility, appreciation, and, potentially, the quality of non-textual research artefacts. The CODECHECK website can be accessed here: https://codecheck.org.uk/.

When Should Data and Code be Made Available?

“…To alleviate the burden, Nüst and Eglen introduce so-called "codecheckers" as potential missing links in the peer review process. 2 Codecheckers are supposed to work alongside peer reviewers and be in direct contact with authors to ensure that all required information is available to reproduce the results presented in a paper.…”

“…But how can readers of preprints and reviewers of submitted papers check that the claims made are computationally reproducible if data and code are not available at the time? Without data and code, they face the “inverse problem in reproducible research”: 2 the impossibility of reconstructing data and code based on the manuscript alone. Figure 1 illustrates this problem and suggests a solution: all research data needed to reproduce the results should be shared at the submission stage 2–4 .…”

“…Without data and code, they face the “inverse problem in reproducible research”: 2 the impossibility of reconstructing data and code based on the manuscript alone. Figure 1 illustrates this problem and suggests a solution: all research data needed to reproduce the results should be shared at the submission stage 2–4 . Statisticians and data scientists have the skills and tools to lead by example – both as authors and reviewers – but they still have to overcome some hurdles.…”

“…Adapted from Figure 1 in Nüst and Eglen. 2 In the cloud on the left (1) the research material is represented: the authors pre-process the raw data to make it usable, decide on software to use, write and rewrite code, and produce intermediate results. But they only submit a manuscript for review as in (2), with summary statistics and aggregated data, and they might only share some of the results.…”

Responsible handling of ethics in data publication

“…We understand that peer review is already a time-consuming process, but because the findings in the article rely on the data, it must be part of the review process. Some journals have introduced specialized review steps [ 8 ], and there are also initiatives to support the review of code [ 9 ]. The Lancet recently implemented specific requirements for papers involving large datasets [ 10 ] including that for those submissions, at least 1 reviewer should have expertise in the dataset reported.…”