Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition

Abstract

Access to data is a critical feature of an efficient, progressive and ultimately self-correcting scientific ecosystem. But the extent to which in-principle benefits of data sharing are realized in practice is unclear. Crucially, it is largely unknown whether published findings can be reproduced by repeating reported analyses upon shared data (‘analytic reproducibility’). To investigate this, we conducted an observational evaluation of a mandatory open data policy introduced at the journal Cognition. Interrupted time-series analyses indicated a substantial post-policy increase in data available statements (104/417, 25% pre-policy to 136/174, 78% post-policy), although not all data appeared reusable (23/104, 22% pre-policy to 85/136, 62%, post-policy). For 35 of the articles determined to have reusable data, we attempted to reproduce 1324 target values. Ultimately, 64 values could not be reproduced within a 10% margin of error. For 22 articles all target values were reproduced, but 11 of these required author assistance. For 13 articles at least one value could not be reproduced despite author assistance. Importantly, there were no clear indications that original conclusions were seriously impacted. Mandatory open data policies can increase the frequency and quality of data sharing. However, suboptimal data curation, unclear analysis specification and reporting errors can impede analytic reproducibility, undermining the utility of data sharing and the credibility of scientific findings.

Advancing Transparency and Openness in Child Development Research: Opportunities

“…Data sharing also provides opportunities for science to correct itself by allowing independent review of data analysis (e.g., Hardwicke et al, 2018). Finding errors in published work is painful, as many researchers know from personal experience.…”

What’s in a looking time preference?

“…Because p values are not always reported exactly, and because there might be errors in computing p values (see Nuijten et al, 2016;Hardwicke et al, 2018), we re-compute p values when possible from test statistics and degrees of freedom extracted from the source paper (i.e. F scores, t -values, etc).…”

Can Orthopaedics become the Gold Standard for Reproducibility? A Roadmap to Success

“…For example, in a study of 21 orthopaedic surgery journals, Checketts et al reported that 52% required clinical trial registration 10 . Furthermore, several studies have examined the implementation of data-sharing policies, finding that their enactment led to an increase in the publications that included raw data 11,12 . With publication retractions in orthopaedics increasing, primarily due to academic misconduct and fraud 13 , the need to verify trials by reproducing studies has become more urgent.…”