Analytic reproducibility in articles receiving open data badges at the journal Psychological Science : an observational study

Abstract

For any scientific report, repeating the original analyses upon the original data should yield the original outcomes. We evaluated analytic reproducibility in 25 Psychological Science articles awarded open data badges between 2014 and 2015. Initially, 16 (64%, 95% confidence interval [43,81]) articles contained at least one ‘major numerical discrepancy' (>10% difference) prompting us to request input from original authors. Ultimately, target values were reproducible without author involvement for 9 (36% [20,59]) articles; reproducible with author involvement for 6 (24% [8,47]) articles; not fully reproducible with no substantive author response for 3 (12% [0,35]) articles; and not fully reproducible despite author involvement for 7 (28% [12,51]) articles. Overall, 37 major numerical discrepancies remained out of 789 checked values (5% [3,6]), but original conclusions did not appear affected. Non-reproducibility was primarily caused by unclear reporting of analytic procedures. These results highlight that open data alone is not sufficient to ensure analytic reproducibility.

Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)

“…Second, although the focus of this study was transparency and reproducibility-related practices, this does not imply that the adoption of these practices is sufficient to promote the goals they are intended to achieve. For example, poorly documented data may not enable analytic reproducibility (Hardwicke, Bohn, et al, 2020;Hardwicke et al, 2018), and inadequately specified preregistrations may not sufficiently constrain researcher degrees of freedom (Claesen et al, 2019;Bakker et al, 2020). Third, we relied only on published information.…”

Share the code, not just the data: A case study of the reproducibility of articles published in the Journal of Memory and Language under the open data policy

“…who assessed the reproducibility of papers published in Cognition, estimated a reproducibility rate of 31% without author assistance, and an additional increase of 31% when the authors of the original manuscript helped to reproduce the outcomes. Similarly,Hardwicke et al (2021) report a reproducibility rate of 36% without author assistance, and an increase of 24% with author assistance for the articles published in Psychological Science that received an 'open data badge'. Similar reproducibility rates were reported for political science…”

Dissecting the tension of open science standards implementation in management and organization journals

“…Data, code, materials, and design and analysis transparency standards require that the respective elements are shared with the readers, thereby allowing for thorough checking and reproducibility of research results. Even though data and code availability do not always ensure perfect reproducibility (Hardwicke et al 2021;Chang and Li 2015), the ability to rerun the code on data increases trust in the results. Furthermore, the ability to inspect all the materials and raw results allows for independent evaluation of threats to external validity (Shadish, Campbell, and Cook 2001), such as under-or over-representation of certain demographics or specifics of materials used for intervention.…”