A practical solution to the pervasive problems ofp values

Abstract

). The issues that have dominated the NHST discussion in the psychological literature are that (1) NHST tempts the user into confusing the probability of the hypothesis given the data with the probability of the data given the hypothesis; (2) .05 is an arbitrary criterion for significance; and (3) in realworld applications, the null hypothesis is never exactly true, and will therefore always be rejected as the number of observations grows large.In the statistical literature, the pros and cons of NHST are also the topic of an ongoing dispute (e.g., Berger & Wolpert, 1988;O'Hagan & Forster, 2004;Royall, 1997;Sellke, Bayarri, & Berger, 2001;Stuart, Ord, & Arnold, 1999). 1 A comparison of these two literatures shows that in psychology, the NHST discussion has focused mostly on problems of interpretation, whereas in statistics, the NHST discussion has focused mostly on problems of formal construction. The statistical perspective on the problems associated with NHST is therefore fundamentally different from the psychological perspective. In this article, the goal is to explain NHST and its problems from a statistical perspective. Many psychologists are oblivious to certain statistical problems associated with NHST, and the examples below show that this ignorance can have important ramifications.In this article, I will show that an NHST p value depends on data that were never observed: The p value is a tail-area integral, and this integral is effectively over data that are not observed but only hypothesized. The probability of these hypothesized data depends crucially on the possibly unknown subjective intentions of the researcher who carried out the experiment. If these intentions were to be ignored, a user of NHST could always obtain a significant result through optional stopping (i.e., analyzing the data as they accumulate and stopping the experiment whenever the p value reaches some desired significance level). In the context of NHST, it is therefore necessary to know the subjective intention with which an experiment was carried out. This key requirement is unattainable in a practical sense, and arguably undesirable in a philosophical sense. In addition, I will review a proof that the NHST p value does not measure statistical evidence. In order for the p value to qualify as a measure of statistical evidence, a minimum requirement is that identical p values convey identical levels of evidence, irrespective 779Copyright 2007 Psychonomic Society, Inc. THEORETICAL AND REVIEW ARTICLESA practical solution to the pervasive problems of p values ERIC-JAN WAGENMAKERS University of Amsterdam, Amsterdam, The NetherlandsIn the field of psychology, the practice of p value null-hypothesis testing is as widespread as ever. Despite this popularity, or perhaps because of it, most psychologists are not aware of the statistical peculiarities of the p value procedure. In particular, p values are based on data that were never observed, and these hypothetical data are themselves influenced by subjective intentions. Moreover, p values d...

Using Bayes factors for testing hypotheses about intervention effectiveness in addictions research

“…Lee & Wagenmakers changed it to moderate, as they thought the original label sounded too decisive 3, 11.…”

Bayesian hypothesis testing for human threat conditioning research: an introduction and the condir R package

“…As mentioned above, however, this statistical approach conveys serious shortcomings (see Cohen, 1994; Krueger, 2001; Krypotos, Blanken, Arnaudova, Matzke, & Beckers, in press; Loftus, 1996; Trafimow, 2003; Wasserstein & Lazar, 2016; Wagenmakers, 2007 for extensive discussions). To name a few within a NHST, it is hard to quantify evidence for the null hypothesis, the stopping plan influences the direction of the results (i.e.…”

“…To name a few within a NHST, it is hard to quantify evidence for the null hypothesis, the stopping plan influences the direction of the results (i.e. whether p -values cross the level), and inferences are based on unobserved data (Cohen, 1994; Wagenmakers, 2007). Given those limitations, many have suggested alternative approaches such as model selection (Tibshirani, 1996; Yuan & Lin, 2006) or Bayesian statistics (Dienes, 2016; Wagenmakers, Morey, & Lee, 2016; Wagenmakers et al, 2017).…”

“…Despite its popularity, this statistical approach has long been heavily criticized, with many pointing to its serious shortcomings (e.g. Cohen, 1994; Wagenmakers, 2007). …”

Reversing the Manual Digit Bias in Two-Digit Number Comparison

“…First, a lower α lessens the risk of a false-positive, or Type I error. Second, p-values between 0.01 and 0.05 often carry small evidential value (Wagenmakers, 2007), particularly when p-values are between 0.04 and 0.05. In such cases, the likelihood of the null hypothesis can actually far outweigh the likelihood of the alternative, a finding known as Lindley's paradox (Lindley, 1957).…”