Data Access, Transparency, and Replication: New Insights from the Political Behavior Literature

Abstract

Do researchers share their quantitative data and are the quantitative results that are published in political science journals replicable? We attempt to answer these questions by analyzing all articles published in the 2015 issues of three political behaviorist journals (i.e., Electoral Studies, Party Politics, and Journal of Elections, Public Opinion & Parties)—all of which did not have a binding data-sharing and replication policy as of 2015. We found that authors are still reluctant to share their data; only slightly more than half of the authors in these journals do so. For those who share their data, we mainly confirmed the initial results reported in the respective articles in roughly 70% of the times. Only roughly 5% of the articles yielded significantly different results from those reported in the publication. However, we also found that roughly 25% of the articles organized the data and/or code so poorly that replication was impossible.

Analysis of Open Data and Computational Reproducibility in Registered Reports in Psychology

“…We limited our analysis to Registered Reports based on the idea that these article formats might be used by people who are early adopters of innovations in science, and would therefore be more likely to also share data and code. We found that the rate at which data and code was shared in our sample was high, relative to studies anlayzed in other reproducibility-focused reviews (e.g, Hardwicke et al, 2018, Stockemer et al, 2018), but we do not have data that gives insights into the motivations of these authors. Registered Reports are written by a diverse set of researchers, working in different subfields in psychology, and it would be interesting for future research to qualitatively examine the motivations of researchers who published Registered Reports for sharing or not sharing data and code.…”

“…The authors estimated that it took between 2-25 hours to reproduce the reported results per article, but they did not record the exact time. Stockemer, Koehler, and Lentz (2018) analyzed reproducibility in all articles published in 2015 in three political science journals. They emailed authors for the code and data, which they received for 71 articles.…”

“…In total, 36 out of 62 (or 58.10%) of the articles shared the underlying data and the code that was used to generate the results. Authors of Registered Reports in psychology seem to share data and code relatively often compared to authors of articles in political science (Stockemer et al, 2018). Compared to authors of non-registered reports in psychology, data and code sharing is also relatively high, as is the reproducibility rate (taking into account that we reproduced articles without contacting the original authors).…”

Successes and struggles with computational reproducibility: Lessons from the Fragile Families Challenge

“…Second, even if the data and code are available, independent researchers are often not able to reproduce the results. These two patterns have been empirically established in the social sciences (Dewald et al, 1986;McCullough et al, 2006;Wicherts et al, 2006;In'nami and Koizumi, 2010;Wicherts et al, 2011;Vanpaemel et al, 2015;Wicherts and Crompvoets, 2017;Gertler et al, 2018;Chang and Li, 2018;Stockemer et al, 2018;Wood et al, 2018), as well as other scientific fields (Savage and Vickers, 2009;Ioannidis et al, 2009;Vandewalle et al, 2009;Gilbert et al, 2012;Ostermann and Granell, 2017;Collberg and Proebsting, 2016;Andrew et al, 2015;Stodden et al, 2018;Konkol et al, 2019;Hardwicke and Ioannidis, 2018;Alsheikh-Ali et al, 2011;Vines et al, 2014;Rowhani-Farid and Barnett, 2016;Naudet et al, 2018;Campbell et al, 2019). Although the magnitude of the problem differs by field and across time, the limited availability of replication materials and the limited usefulness of the materials that are available are consistent patterns.…”

Improving Social Science: Lessons from the Open Science Movement

“…Moreover, while taste for novelty is arguably less of an issue than in psychology, political desirability can have similar influence (Zigerell 2017). And certain problems that have been identified in psychology have also been pointed out in political science, including low computational reproducibility (Stockemer, Koehler, and Lentz 2018; cf. Jacoby, Lafferty-Hess and Christian 2017) and sanitized research narratives that do not capture the actual complexity of the process (Yom 2018).…”