The State of Reproducible Research in Computer Science

Reproducibility in Evolutionary Computation

“…Intellectual property. Concerns about licensing, privacy and commercially sensitive information may be legitimate obstacles for making artifacts (source code or data) publicly available [Fonseca Cacho and Taghva 2020]. Although it may be tempting to make artifacts available only to reviewers under some type of nondisclosure agreement [Heroux 2015;Stodden et al 2016], such an arrangement does not actually improve reproducibility.…”

“…Although this case may seem similar to the availability of sensitive artifacts discussed above, where we argued against making sensitive artifacts only available to reviewers, there is a fundamental difference: Artifacts that are only disclosed to reviewers will never become available to other researchers, which hinders reproducibility, whereas specialist hardware such as GPUs is publicly available for purchase by interested researchers but reviewers should not bear the cost. A similar distinction may be made between undisclosed data, which is not suitable for reproducibility, and data that is simply too large to host or copy for review purposes [Fonseca Cacho and Taghva 2020]. Journals might consider making resources available to their reviewers and bear some of the cost.…”

CG‐SENSE revisited: Results from the first ISMRM reproducibility challenge

“…However, comparing novel algorithms to re‐implementations of published work without having access to the original code can lead to wrong conclusions. Often, algorithmic details are not reported in detail in publications and small deviations of input parameters can lead to strong differences in the output, regularly degrading the performance of the existing method, which is a general problem faced in the scientific community 2‐8 . A questionnaire opened by the RRSGG (https://blog.ismrm.org/2019/04/15/reproducible-research-study-group-questionnaire/), regarding reproducible research, also showed that the majority of the 71 participants (77.5%) sees a reproducibility problem in their research area.…”

“…community. [2][3][4][5][6][7][8] A questionnaire opened by the RRSGG (https:// blog.ismrm.org/2019/04/15/repro ducib le-resea rch-study -groupquest ionna ire/), regarding reproducible research, also showed that the majority of the 71 participants (77.5%) sees a reproducibility problem in their research area. This proves that scientists are aware of the problem of reproducibility of research and how hard it can be to recreate paper results without access to code or data.…”

CG-SENSE revisited: Results from the first ISMRM reproducibility challenge

“…However, comparing novel algorithms to re-implementations of published work without having access to the original code can lead to wrong conclusions. Often, algorithmic details are not reported in detail in publications and small deviations of input parameters can lead to strong differences in the output, regularly degrading the performance of the existing method, which is a general problem faced in the scientific community [2,3,4,5,6,7,8]. To that end, the RRSG announced a reproducibility challenge in April 2019 as part of the Annual Meeting of the ISMRM in Montreal.…”