Operationalizing the Replication Standard: A Case Study of the Data Curation and Verification Workflow for Scholarly Journals

Abstract

In response to widespread concerns about the integrity of research published in scholarly journals, several initiatives have emerged that are promoting research transparency through access to data underlying published scientific findings. Journal editors, in particular, have made a commitment to research transparency by issuing data policies that require authors to submit their data, code, and documentation to data repositories to allow for public access to the data. In the case of the American Journal of Political Science (AJPS) Data Replication Policy, the data also must undergo an independent verification process in which materials are reviewed for quality as a condition of final manuscript publication and acceptance.Aware of the specialized expertise of the data archives, AJPS called upon the Odum Institute Data Archive to provide a data review service that performs data curation and verification of replication datasets. This article presents a case study of the collaboration between AJPS and the Odum Institute Data Archive to develop a workflow that bridges manuscript publication and data review processes. The case study describes the challenges and the successes of the workflow integration, and offers lessons learned that may be applied by other data archives that are considering expanding their services to include data curation and verification services to support reproducible research.

Sharing is Caring: Ethical Implications of Transparent Research in Psychology

“…Since the majority of journals do not require data archiving as part of the publication process, researchers are independently responsible to maintain their own data archives (Nosek et al, 2021). Sharing data often requires considerable work and preparation (Alter & Gonzalez, 2018;Christian et al, 2018;Nosek et al, 2021). The researchers who did respond to the data requests by Wicherts et al (2006) reported that providing manageable data and a workable codebook took substantial time and effort.…”

Toward a FAIR Reproducible Research

“…Until very recently, very few (e.g., JSS, AJPS) checked that the code runs and reproduces the key results (Christian et al, 2018). Moreover, Duvendack et al (2017) showed that in only 28 economic journals (out of 333) do a majority of the empirical papers supply their data and code.…”

Report by the AEA Data Editor

“…A data editor with a mandate similar to mine was recently appointed at the Review of Economic Studies. In political science, the American Journal of Political Science in collaboration with the Odum Institute for Research in Social Science (Christian et al 2018) has been conducting data curation and code verification. The Journal of the American Statistical Association has been performing a "broad evaluation of quality and potential for usability of the code and data" since 2016 (Stodden et al 2016).…”

“…To that end, we will reinforce the data and code availability policies above by conducting pre-publication verification of code reproducibility. To our knowledge, this has been systematically done only at the American Journal of Political Science (Christian et al 2018), though the AJPS also does significant data curation tasks for the authors. The sizable challenge is how to conduct such pre-publication verification without unduly delaying the editorial work flow.…”

“…The sizable challenge is how to conduct such pre-publication verification without unduly delaying the editorial work flow. Christian et al (2018) report lengthy delays between acceptance and publication due to the combination of data curation and code verification. Several factors lead us to think that this is feasible.…”