The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research

Abstract

The widespread use of ‘statistical significance’ as a license for making a claim of a scientific finding leads to considerable distortion of the scientific process (according to the American Statistical Association). We review why degrading p-values into ‘significant’ and ‘nonsignificant’ contributes to making studies irreproducible, or to making them seem irreproducible. A major problem is that we tend to take small p-values at face value, but mistrust results with larger p-values. In either case, p-values tell little about reliability of research, because they are hardly replicable even if an alternative hypothesis is true. Also significance (p ≤ 0.05) is hardly replicable: at a good statistical power of 80%, two studies will be ‘conflicting’, meaning that one is significant and the other is not, in one third of the cases if there is a true effect. A replication can therefore not be interpreted as having failed only because it is nonsignificant. Many apparent replication failures may thus reflect faulty judgment based on significance thresholds rather than a crisis of unreplicable research. Reliable conclusions on replicability and practical importance of a finding can only be drawn using cumulative evidence from multiple independent studies. However, applying significance thresholds makes cumulative knowledge unreliable. One reason is that with anything but ideal statistical power, significant effect sizes will be biased upwards. Interpreting inflated significant results while ignoring nonsignificant results will thus lead to wrong conclusions. But current incentives to hunt for significance lead to selective reporting and to publication bias against nonsignificant findings. Data dredging, p-hacking, and publication bias should be addressed by removing fixed significance thresholds. Consistent with the recommendations of the late Ronald Fisher, p-values should be interpreted as graded measures of the strength of evidence against the null hypothesis. Also larger p-values offer some evidence against the null hypothesis, and they cannot be interpreted as supporting the null hypothesis, falsely concluding that ‘there is no effect’. Information on possible true effect sizes that are compatible with the data must be obtained from the point estimate, e.g., from a sample average, and from the interval estimate, such as a confidence interval. We review how confusion about interpretation of larger p-values can be traced back to historical disputes among the founders of modern statistics. We further discuss potential arguments against removing significance thresholds, for example that decision rules should rather be more stringent, that sample sizes could decrease, or that p-values should better be completely abandoned. We conclude that whatever method of statistical inference we use, dichotomous threshold thinking must give way to non-automated informed judgment.

Estimating volumes from common carp hepatocytes using design‐based stereology and examining correlations with profile areas: Revisiting a nutritional assay and unveiling guidelines to microscopists

“…The paired Student t tests and the correlation analyses were made not only with data derived from this study, but also recurring to the original data sets derived from the same fish, as provided by the authors (Poleksić et al, ). As a measure of graded evidence (Amrhein, Korner‐Nievergelt, & Roth, ), the significance level (α) was set at 5%.…”

Presidential address: Six open questions to genetic epidemiologists

“…Interestingly, there is a Bayesian line of argumentation to justify the LOD score threshold of 3, showing that under a number of assumptions, this threshold yields a false-positive frequency of about 5% (see also Khoury, Beaty, & Cohen, 1993). Another recommendation is not to rely on thresholds as such but to use and interpret p values differently as "graded measures of the strength of evidence against H 0 " KÖNIG | 243 (Fisher, 1956) or as "measure of surprise: the smaller it is, the more surprising the results are if H 0 is true" (Amrhein, Korner-Nievergelt, & Roth, 2017). Another recommendation is not to rely on thresholds as such but to use and interpret p values differently as "graded measures of the strength of evidence against H 0 " KÖNIG | 243 (Fisher, 1956) or as "measure of surprise: the smaller it is, the more surprising the results are if H 0 is true" (Amrhein, Korner-Nievergelt, & Roth, 2017).…”

An Evaluation of Four Solutions to the Forking Paths Problem: Adjusted Alpha, Preregistration, Sensitivity Analyses, and Abandoning the Neyman-Pearson Approach

“…Unlike the Neyman-Pearson approach, Fisherian p values are interpreted on a sliding scale of probability in which the smaller the p value, the less likely it is that the observed data would occur if the null hypothesis was true (Amrhein, Korner-Nievergelt, & Roth, 2017;Biau, Jolles, & Porcher, 2010;Bradley & Brand, 2016;Falissard, 2012;Haig, 2016;Hubbard & Bayarri, 2003;Wasserstein & Lazar, 2016). Hence, from a Fisherian perspective, p = .0004 indicates stronger evidence against the null hypothesis than p = .04 (Hubbard & Lindsay, 2008).…”

“…The typical method of conducting null hypothesis significance tests often represents a hybrid of the Neyman-Pearson and Fisherian approaches (Amrhein et al, 2017;Biau et al, 2010;Bradley & Brand, 2016;Hubbard & Bayarri, 2003;Schneider, 2015). I discuss the Fisherian approach later on in this article.…”

“…This context-based approach provides a more nuanced and sophisticated approach to significance testing, in which researchers interpret the meaning of p values based on a range of background factors including (a) the size of the p value, (b) the number of tests that have been used to assess the null hypothesis, (c) the theoretical plausibility of the explanation, (d) prior evidence for the effect, (e) the effect size, (f) the variability of the effect, (g) the quality of the research methodology (e.g., sample size, measurement precision, construct validity, etc. ), (h) the results of sensitivity analyses, and (i) the results of tests of alternative explanations for the effect (e.g., Amrhein et al, 2017;Wasserstein & Lazar 2016). For example, researchers should interpret a p value of .025 differently when it is used to infer the existence of extrasensory perception compared to when it is used to infer the existence of ethnic prejudice.…”