How to make replication the norm

Gertler, Paul; Galiani, Sebastián; Romero, Mauricio

doi:10.1038/d41586-018-02108-9

Cited by 68 publications

(43 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Despite these advantages, publishing results in a reproducible way is still not common practice (Reichman et al 2011), which is part of the reason why some have proclaimed a 'reproducibility crisis' (Baker 2016). A recent study in economics (Gertler et al 2018) has shown that even when authors make the data and code publicly accessible, it is not guaranteed that readers can successfully reproduce the results published in the paper. On top of that, the inconsistent usage of the terms reproducibility and replicability within and across disciplines can cause further CONTACT Markus Konkol m.konkol@uni-muenster.de Supplemental data for this article can be accessed here.…”

Section: Introductionmentioning

confidence: 99%

Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study

Konkol

Kray

Pfeiffer

2018

International Journal of Geographical Information Science

View full text Add to dashboard Cite

Reproducibility is a cornerstone of science and thus for geographic research as well. However, studies in other disciplines such as biology have shown that published work is rarely reproducible. To assess the state of reproducibility, specifically computational reproducibility (i.e. rerunning the analysis of a paper using the original code), in geographic research, we asked geoscientists about this topic using three methods: a survey (n = 146), interviews (n = 9), and a focus group (n = 5). We asked participants about their understanding of open reproducible research (ORR), how much it is practiced, and what obstacles hinder ORR. We found that participants had different understandings of ORR and that there are several obstacles for authors and readers (e.g. effort, lack of openness). Then, in order to complement the subjective feedback from the participants, we tried to reproduce the results of papers that use spatial statistics to address problems in the geosciences. We selected 41 open access papers from Copernicus and Journal of Statistical Software and executed the R code. In doing so, we identified several technical issues and specific issues with the reproduced figures depicting the results. Based on these findings, we propose guidelines for authors to overcome the issues around reproducibility in the computational geosciences. ARTICLE HISTORY

show abstract

Section: Introductionmentioning

confidence: 99%

Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study

Konkol

Kray

Pfeiffer

2018

International Journal of Geographical Information Science

View full text Add to dashboard Cite

show abstract

“…Second, even if the data and code are available, independent researchers are often not able to reproduce the results. These two patterns have been empirically established in the social sciences (Dewald et al, 1986;McCullough et al, 2006;Wicherts et al, 2006;In'nami and Koizumi, 2010;Wicherts et al, 2011;Vanpaemel et al, 2015;Wicherts and Crompvoets, 2017;Gertler et al, 2018;Chang and Li, 2018;Stockemer et al, 2018;Wood et al, 2018), as well as other scientific fields (Savage and Vickers, 2009;Ioannidis et al, 2009;Vandewalle et al, 2009;Gilbert et al, 2012;Ostermann and Granell, 2017;Collberg and Proebsting, 2016;Andrew et al, 2015;Stodden et al, 2018;Konkol et al, 2019;Hardwicke and Ioannidis, 2018;Alsheikh-Ali et al, 2011;Vines et al, 2014;Rowhani-Farid and Barnett, 2016;Naudet et al, 2018;Campbell et al, 2019). Although the magnitude of the problem differs by field and across time, the limited availability of replication materials and the limited usefulness of the materials that are available are consistent patterns.…”

Section: Introductionmentioning

confidence: 94%

Successes and struggles with computational reproducibility: Lessons from the Fragile Families Challenge

Liu¹,

Salganik²

2019

Preprint

View full text Add to dashboard Cite

Reproducibility is fundamental to science, and an important component of reproducibility is computational reproducibility: the ability of a researcher to recreate the results in a published paper using the original author's raw data and code. Although most people agree that computational reproducibility is important, it is still difficult to achieve in practice. In this paper, we describe our approach to enabling computational reproducibility for the 12 papers in this special issue of Socius about the Fragile Families Challenge. Our approach draws on two tools commonly used by professional software engineers but not widely used by academic researchers: software containers (e.g., Docker) and cloud computing (e.g., Amazon Web Services). These tools enabled us to standardize the computing environment around each submission, which will ease computational reproducibility both today and in the future. Drawing on our successes and struggles, we conclude with recommendations to authors and journals.

show abstract

“…Brown and Wood note that the replication program at 3ie first selects candidate studies for replication, then matches researchers to do the replication. Nevertheless, this suspicion is difficult to shake given the predilection of journals to publish replications that overturn the results of an original study, thus incentivizing replication authors to find faults (Gertler, Galiani, and Romero, 2018). This motivates the last takeaway.…”

Section: Takeawaysmentioning

confidence: 99%

Takeaways from the Special Issue on The Practice of Replication

Reed

2019

Economics

View full text Add to dashboard Cite

In July 2017, Economics: The Open Access, Open Assessment E-Journal issued a call for papers for a special issue on "The Practice of Replication." In that call, the journal explained that there was no generally accepted procedure for how to do a replication. Likewise, there was no generally accepted standard for determining whether a replication "confirms or disconfirms" an original study. Accordingly, the journal called for papers to identify principles for how to do a replication and how to interpret its results; and to apply those principles in crafting a replication plan for a study of the author's choosing. The hope was that this exercise would produce some progress on "the practice of replication." The special issue is now complete with a total of eight journal articles. This commentary places the respective articles within a common framework and identifies observations and lessons learned from the respective studies.

show abstract

How to make replication the norm

Cited by 68 publications

References 8 publications

Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study

Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study

Successes and struggles with computational reproducibility: Lessons from the Fragile Families Challenge

Takeaways from the Special Issue on The Practice of Replication

Contact Info

Product

Resources

About