The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline

Abstract

This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had "in the pipeline" as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

“…In the field of education, Makel and Plucker (2014) found a publication rate of 0.13% for replication studies (221 out of 164,589 articles) in the 100 highest-impact journals between 1938 and 2014. In the field of psychology, Makel et al (2012) estimated that among the top 100 journals between 1900 and 2010, the replication study publication rate was 1.07%, though this rate is now likely to be higher given recent multiple, direct replication projects: the Many Labs project (Klein et al, 2014), the Pipeline Project (Schweinsberg et al, 2016), the Registered Reports project (Nosek & Lakens, 2014), and the Reproducibility Project (Open Science Collaboration, 2015). In business, marketing, and communication journals, replication rates have ranged from 1 to 3% (Evanschitzky, Baumgarth, Hubbard, & Armstrong, 2007;Hubbard & Armstrong, 1994;Kelly, Chase, & Tucker, 1979).…”

Contextual sensitivity in scientific reproducibility

“…† The authors concluded that context (i.e., sample/setting) had "little systematic effect on the observed results" (51). In contrast, a project examining the reproducibility of 10 effects related to moral judgment (seven were reproduced consistently and one was reproduced weakly) across 25 international samples (62) found evidence that certain effects were reproducible only within the culture in which they were originally observed. In other words, context moderated replication success.…”

“…Numerous other suggestions for improving reproducibility have been proposed (e.g., refs. 62,76,86,87). For example, the replication recipe (76) offers a "five-ingredient" approach to standardizing replication attempts that emphasizes precision, power, transparency, and collaboration.…”

Replication in Experimental Economics: A Historical and Quantitative Approach Focused on Public Good Game Experiments

“…To hedge against this potential bias, some authors argue that studies should be accepted or rejected before the results are actually being produced (e.g., Zhang and Ortmann, 2013;Dufwenberg, 2015). This method is currently explored in other disciplines (e.g., in social psychology, Schweinsberg et al, 2016). The last family of explanations for the decline effect is in terms of statistical biases.…”