A large-scale study on research code quality and execution

Abstract

This article presents a study on the quality and execution of research code from publicly-available replication datasets at the Harvard Dataverse repository. Research code is typically created by a group of scientists and published together with academic papers to facilitate research transparency and reproducibility. For this study, we define ten questions to address aspects impacting research reproducibility and reuse. First, we retrieve and analyze more than 2000 replication datasets with over 9000 unique R files published from 2010 to 2020. Second, we execute the code in a clean runtime environment to assess its ease of reuse. Common coding errors were identified, and some of them were solved with automatic code cleaning to aid code execution. We find that 74% of R files failed to complete without error in the initial execution, while 56% failed when code cleaning was applied, showing that many errors can be prevented with good coding practices. We also analyze the replication datasets from journals’ collections and discuss the impact of the journal policy strictness on the code re-execution rate. Finally, based on our results, we propose a set of recommendations for code dissemination aimed at researchers, journals, and repositories.

What’s in a Badge? A Computational Reproducibility Investigation of the Open Data Badge Policy in one Issue of Psychological Science

“…While the former clearly do not meet the eligibility requirements, we cannot judge the reproducibility, and thus the badge eligibility, of the latter at submission. We make several recommendations for how to improve the specific badge policy at Psychological Science and comparable initiatives at other journals (for further general recommendations on how to improve data sharing and computational reproducibility, see, e.g., Stodden et al, 2016;Trisovic et al, 2022;Wilson et al, 2017). Excellent and more in-depth recommendations and tutorials for authors to ensure that their shared data and code are eligible for an Open Data badge are provided by, for example, Arslan (2019), Eberle ( 2022 2021).…”

The Dire Disregard of Measurement Invariance Testing in Psychological Science

“…However, especially in structural equation modeling research, sharing syntax is essential to evaluate model specifications and decisions, as they are often unavailable upon request (Wicherts & Crompvoets, 2017). In addition, many minor adjustments, such as capturing research workflow or managing project library paths and dependencies, can significantly improve the quality of research code (Trisovic et al, 2022). On the other hand, it is up to journals to ensure that authors with an open data statement in their manuscript share their (intelligible) data.…”

CACP: Classification Algorithms Comparison Pipeline

“…First, the literature review indicates that the source code is often unavailable, making it impossible to reproduce the study accurately. Second, the algorithms are published in pseudo-code, but its informality risks overlooked errors or new errors being introduced when translated to real programming languages [13,14]. Additionally, they were tested on a small number of datasets [15][16][17].…”