The case for open computer programs

Abstract

Scientific communication relies on evidence that cannot be entirely included in publications, but the rise of computational science has added a new layer of inaccessibility. Although it is now accepted that data should be made available on request, the current regulations regarding the availability of software are inconsistent. We argue that, with some exceptions, anything less than the release of source programs is intolerable for results that depend on computation. The vagaries of hardware, software and natural language will always ensure that exact reproducibility remains uncertain, but withholding code increases the chances that efforts to reproduce results will fail.T he rise of computational science has led to unprecedented opportunities for scientific advance. Ever more powerful computers enable theories to be investigated that were thought almost intractable a decade ago, robust hardware technologies allow data collection in the most inhospitable environments, more data are collected, and an increasingly rich set of software tools are now available with which to analyse computer-generated data.However, there is the difficulty of reproducibility, by which we mean the reproduction of a scientific paper's central finding, rather than exact replication of each specific numerical result down to several decimal places. We examine the problem of reproducibility (for an early attempt at solving it, see ref. 1) in the context of openly available computer programs, or code. Our view is that we have reached the point that, with some exceptions, anything less than release of actual source code is an indefensible approach for any scientific results that depend on computation, because not releasing such code raises needless, and needlessly confusing, roadblocks to reproducibility.At present, debate rages on the need to release computer programs associated with scientific experiments 2-4 , with policies still ranging from mandatory total release to the release only of natural language descriptions, that is, written descriptions of computer program algorithms. Some journals have already changed their policies on computer program openness; Science, for example, now includes code in the list of items that should be supplied by an author 5 . Other journals promoting code availability include Geoscientific Model Development, which is devoted, at least in part, to model description and code publication, and Biostatistics, which has appointed an editor to assess the reproducibility of the software and data associated with an article 6 .In contrast, less stringent policies are exemplified by statements such as 7 ''Nature does not require authors to make code available, but we do expect a description detailed enough to allow others to write their own code to do similar analysis.'' Although Nature's broader policy states that ''...authors are required to make materials, data and associated protocols promptly available to readers...'', and editors and referees are fully empowered to demand and evaluate any specific code, we belie...

Do you speak open science? Resources and tips to learn the language

“…The case for open source code is straightforward: the code researchers write and use to analyze data is a vital part of the scientific research cycle, and, similar to data, is not only necessary to reproduce and interpret the results and corresponding conclusions, but can also be used to answer novel research questions. Therefore, if researchers write code as a means to obtain results from data, then this code should be released as well [8]. Clear arrangements for the storage and preservation of the code should be made, instructions need to be provided that will allow the code to be compiled and run without issue, and the code should be accompanied by a description of the core functionalities and hard-and software requirements for its use.…”

Agent-Based Modeling in Systems Pharmacology

“…Principled design frameworks that aim to enforce a robust model development methodology facilitate these goals. Grimm et al 45. proposed a “three‐block” standard protocol for describing ABMs termed ODD (overview, design concepts, and details).…”

Cloud–Based Evaluation Framework for Big Data

“…Current evaluation campaigns include TREC (Text REtrieval Conference) 5 , TRECVid (TREC Video Evaluation) 6 , CLEF (Cross Language Evaluation Forum) 7 , ImageCLEF [14], NTCIR (NII Test Collection for IR Systems) 8 , INEX (Initiative for the Evaluation of XML Retrieval) 9 and FIRE (Forum for Information Retrieval Evaluation) 10 . In the area of machine learning, the PAS-CAL challenges are well known 11 , while in the area of medical image analysis, annual challenges are organised as part of the Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 12 . However, even with these evaluation campaigns and challenges, a number of causes contribute to the above-mentioned lack of clear improvement:…”

“…The lack of publication of program code has been identified as a significant reason for this [6]. There is currently work underway to counter this situation, ranging from presenting the case for open computer programs [12], through creating infrastructures to allow reproducible computational research [6] to considerations about the legal licensing and copyright frameworks for computational research [18].…”