A Bayes factor meta-analysis of Bem’s ESP claim

Abstract

In recent years, statisticians and psychologists have provided the critique that p-values do not capture the evidence afforded by data and are, consequently, ill suited for analysis in scientific endeavors. The issue is particular salient in the assessment of the recent evidence provided for ESP by Bem (2011) in the mainstream Journal of Personality and Social Psychology. Wagenmakers, Wetzels, Borsboom, and van der Maas (Journal of Personality and Social Psychology, 100, 426-432, 2011) have provided an alternative Bayes factor assessment of Bem's data, but their assessment was limited to examining each experiment in isolation. We show here that the variant of the Bayes factor employed by Wagenmakers et al. is inappropriate for making assessments across multiple experiments, and cannot be used to gain an accurate assessment of the total evidence in Bem's data. We develop a meta-analytic Bayes factor that describes how researchers should update their prior beliefs about the odds of hypotheses in light of data across several experiments. We find that the evidence that people can feel the future with neutral and erotic stimuli to be slight, with Bayes factors of 3.23 and 1.57, respectively. There is some evidence, however, for the hypothesis that people can feel the future with emotionally valenced nonerotic stimuli, with a Bayes factor of about 40. Although this value is certainly noteworthy, we believe it is orders of magnitude lower than what is required to overcome appropriate skepticism of ESP.Keywords Statistics . Statistical inference . ESP Bem (2011) has claimed that people can feel or sense salient events in the future that could not otherwise be anticipated. For example, in his Experiment 2, Bem presented participants with two rather ordinary pictures and asked them to indicate which one would be chosen subsequently by a random number generator. If a participant correctly anticipated the random choice, he or she was rewarded with a brief display of a positively valenced picture. Conversely, if a participant incorrectly anticipated the random choice, he or she was punished with a negatively valenced picture. Bem claimed that people could indeed feel these future reward and punishment events and, consequently, were able to anticipate the random choice at a rate deemed statistically above chance. Bem presented a sequence of similar experiments and results and, on this basis, concluded that people can feel the future. This phenomenon and others like it in which people can show seemingly impossible awareness of events are termed psi phenomena, or, more colloquially, extrasensory perception (ESP).If ESP is substantiated, it would be among the most important findings in the history of psychology. The existence of ESP would force us to revise not only our theories of psychology, but also those of biology and physics. In our view, when seemingly implausible claims are made with conventional methods, it provides an ideal moment to reexamine these methods. The conventional approach used by Bem (2011) has two...

Reevaluating the effectiveness of n-back training on transfer through the Bayesian lens: Support for the null

“…As intimated above, a feature of the Bayesian analysis is that it permits one to evaluate the likelihood of the data under both the null hypothesis of no transfer to Gf and the alternative hypothesis that n-back training transfers to Gf. Our analysis approach closely followed the methods used by Rouder and Morey (2011) and Rouder et al (2013) in their meta-analyses of psi (i.e., extrasensory perception). Furthermore, we used only those effect sizes included in Fig.…”

“…The aggregate weighted effect size across these 24 comparisons was 0.24. They also evaluated several possible mediators, including whether the studies used an active control (N = 12) or a passive control 1 The limitations of NHST methods are well documented and need not be rehashed here in their entirety; readers interested in this topic are invited to read papers by Raftery (1995), Wagenmakers (2007), Rouder, Speckman, Sun, Morey, and Iverson (2009), and in particular Rouder and Morey (2011) and Rouder, Morey, and Province (2013). 2 This is particularly challenging for cognitive training studies because participants literally see and engage in the intervention, making it difficult to mask what condition participants believe they have been assigned to.…”

Too good to be true: Publication bias in two prominent studies from experimental psychology

“…The analytical criticisms are also only partially convincing. Although Wagenmakers et al noted that the individual experiments in Bem (2011) do not meet the analytical criteria of a standard Bayesian analysis, Rouder and Morey (2011) used a Bayesian metaanalysis and found some evidence for the proposed psi effect, although the latter authors emphasized that such evidence must be considered in the context of other, conflicting evidence.…”

Introduction to Bayesian Inference for Psychology

“…Marchbanks' inferred prior probabilities of each of the three hypotheses are (25,50,25), Granger's are (15,70,15), and Runcorn's are (5,15,80). This type of procedure is known as prior elicitation; for more in-depth discussion on prior elicitation, see Garthwaite et al (13) and Lee and Vanpaemel (30).…”

“…Consider as an extreme example Bem (1) who presented data consistent with the hypothesis that some humans can predict future random events. While Bem's data may indeed provide positive evidence for that hypothesis (50), it is staggeringly improbable a priori and the evidence in the data does not stack up to the strong priors many of us will have regarding extrasensory perception -extraordinary claims require extraordinary evidence.…”