Making Replication Mainstream

Abstract

Many philosophers of science and methodologists have argued that the ability to repeat studies and obtain similar results is an essential component of science. A finding is elevated from single observation to scientific evidence when the procedures that were used to obtain it can be reproduced and the finding itself can be replicated. Recent replication attempts show that some high profile results--most notably in psychology, but in many other disciplines as well---cannot be replicated consistently. These replication attempts have generated a considerable amount of controversy and the issue of whether direct replications have value has, in particular, proven to be contentious. However, much of this discussion has occurred in published commentaries and social media outlets, resulting in a fragmented discourse. To address the need for an integrative summary, we review various types of replication studies and then discuss the most commonly voiced concerns about direct replication. We provide detailed responses to these concerns and consider different statistical ways to evaluate replications. We conclude there are no theoretical or statistical obstacles to making direct replication a routine aspect of psychological science.

No evidence that economic inequality moderates the effect of income on generosity

“…For instance, the National Academy of Sciences organized a colloquium on this issue (41) and initiated a committee on reproducibility and replicability in December 2017 (42). We support the idea that replications should become the norm rather than the exception before new findings are readily accepted, even when such findings appear to be plausible and desirable (43,44). As the importance of a study increases, it is even more essential to confirm the reproducibility and replicability of that research, and importance might be defined through a study's theoretical weight, societal implications, influence through citations, or mass appeal (45).…”

Robust Modeling in Cognitive Science

“…A new publication format known as registered reports has been adopted by more than 100 psychology journals as a way to incorporate these ideas directly into the research and publication pipeline 1 (Chambers, 2013;Chambers, Dienes, McIntosh, Rotshtein, & Willmes, 2015;Hardwicke & Ioannidis, 2018). Psychologists have also recognized the importance of replication as a tool for verifying scientific claims (Open Science Collaboration, 2012, 2015, and vigorously debated what role replication plays in a healthy science (see Zwaan, Etz, Lucas, & Donnellan, 2018, and its associated commentaries). In addition, psychologists have pushed for open data, open code, and open materials to allow for better verification and reanalysis of study results.…”

Heterogeneity in direct replications in psychology and its association with effect size.

“…In the second stage, one may ask to what extent the effect depends on a particular choice of four contextual factors; the 1) sample population, 2) settings, 3) treatment variables and 4) measurement variables (e.g., Campbell & Stanley, 2015). This extent is often explored through replications of the original study that are either as similar as possible to the original (called 'direct' or 'exact' replications) or with some deliberate variation on conceptual factors (so-called 'conceptual' or 'indirect' replications;Zwaan et al, 2017), and once sufficient studies have accumulated through meta-analysis. In meta-analysis, the heterogeneity of an effect size (henceforth referred to as heterogeneity) is a measure of an effect's susceptibility to changes in these four factors.…”