Correcting the past: Failures to replicate psi.

Abstract

Across 7 experiments (N = 3,289), we replicate the procedure of Experiments 8 and 9 from Bem (2011), which had originally demonstrated retroactive facilitation of recall. We failed to replicate that finding. We further conduct a meta-analysis of all replication attempts of these experiments and find that the average effect size (d = 0.04) is no different from 0. We discuss some reasons for differences between the results in this article and those presented in Bem (2011).

Safeguard Power as a Protection Against Imprecise Power Estimates

“…Recent years have seen an unprecedented interest in the issue of replicability, including several replication studies (e.g., Doyen, Klein, Pichon, & Cleeremans, 2012;Galak, LeBoeuf, Nelson, & Simmons, 2012;LeBel & Campbell, 2013;Shanks et al, 2013), contributions relevant for the broader issue of replicability (Asendorpf et al, 2013;Brandt et al, 2014;Pashler & Wagenmakers, 2012;Simonsohn, 2013), and a large, ongoing, collaborative effort to systematically replicate research findings (Open Science Collaboration, 2012). This interest in replication studies has placed in center stage the issue of power analysis (Cohen, 1992), both as a post hoc calculation of the likelihood of finding a significant effect, if it exists, and as a way to plan the sample size needed to achieve a certain likelihood of finding a significant effect.…”

A Large Scale Test of the Effect of Social Class on Prosocial Behavior

“…As Schmidt [13] put it, "Replication is one of the most important tools for the verification of facts within the empirical sciences" (p. 90; see also [130]). It even helps to advance our theories by showing boundary conditions or moderators that helped to produce the effects that were found in the first place and, thus, provides starting points for future research ( [12,13,131,132]; for some recent concrete examples, e.g., see Donnellan, Lucas, & Cesario [133], on the results of Bargh & Shalev, [134]; or Galak, LeBoeuf, Nelson, & Simmons, [135], and Wagenmakers, Wetzels, Borsboom, & van der Maas, [136], on the results of Bem, [137]; see also the first results of the "many labs" replication project, [138]). …”

Does the conclusion follow from the evidence? Recommendations for improving research

“…For example, multiple analyses have questioned whether findings in support of precognition (Bem, 2011) are too good to be obtained without using questionable research practices (Francis, 2012;Schimmack, 2012). In line with these analyses, researchers who have replicated Bem's procedures have not replicated his results (Galak, LeBoeuf, Nelson, & Simmons, 2012;Ritchie, Wiseman, & French, 2012;Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012). Given the analyses questioning the truth of the original research, failed replications such as these are no longer being disregarded as uninformative.…”

“…Type III and IV errors 4 can be illustrated by considering the differing claims of Bem (2011) and those who have performed failed replications (Galak et al, 2012;Ritchie et al, 2012;Wagenmakers et al, 2012) under different hypothetical assumptions about the null hypothesis. Imagine, for the sake of example, that precognition is a real phenomenon (a false null hypothesis).…”