Reply to Gelman, Gaudart, Pericchi: More reasons to revise standards for statistical evidence

Johnson, Valen E.

doi:10.1073/pnas.1400338111

Cited by 4 publications

(4 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interpreting p -values as graded evidence would mean to mistrust p -values around 0.05, or smaller, depending on the circumstances and the scientific discipline. To announce a discovery in particle physics, a p -value as small as 0.0000003 may be needed ( Johnson, 2014 ). Also in other disciplines, we should often judge our evidence as strong only if the p -value is much smaller than 0.05: if we want to demonstrate a surprising, counterintuitive effect; if we know that our null hypothesis has a high prior probability ( Bayarri et al, 2016 ); if our sample size is large ( Anderson, Burnham & Thompson, 2000 ; Pericchi, Pereira & Perez, 2014 ); or if postulating an effect that in reality is negligible would have serious practical consequences.…”

Section: Introductionmentioning

confidence: 99%

The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research

Amrhein

Korner‐Nievergelt

Roth

2017

PeerJ

266

216

View full text Add to dashboard Cite

The widespread use of ‘statistical significance’ as a license for making a claim of a scientific finding leads to considerable distortion of the scientific process (according to the American Statistical Association). We review why degrading p-values into ‘significant’ and ‘nonsignificant’ contributes to making studies irreproducible, or to making them seem irreproducible. A major problem is that we tend to take small p-values at face value, but mistrust results with larger p-values. In either case, p-values tell little about reliability of research, because they are hardly replicable even if an alternative hypothesis is true. Also significance (p ≤ 0.05) is hardly replicable: at a good statistical power of 80%, two studies will be ‘conflicting’, meaning that one is significant and the other is not, in one third of the cases if there is a true effect. A replication can therefore not be interpreted as having failed only because it is nonsignificant. Many apparent replication failures may thus reflect faulty judgment based on significance thresholds rather than a crisis of unreplicable research. Reliable conclusions on replicability and practical importance of a finding can only be drawn using cumulative evidence from multiple independent studies. However, applying significance thresholds makes cumulative knowledge unreliable. One reason is that with anything but ideal statistical power, significant effect sizes will be biased upwards. Interpreting inflated significant results while ignoring nonsignificant results will thus lead to wrong conclusions. But current incentives to hunt for significance lead to selective reporting and to publication bias against nonsignificant findings. Data dredging, p-hacking, and publication bias should be addressed by removing fixed significance thresholds. Consistent with the recommendations of the late Ronald Fisher, p-values should be interpreted as graded measures of the strength of evidence against the null hypothesis. Also larger p-values offer some evidence against the null hypothesis, and they cannot be interpreted as supporting the null hypothesis, falsely concluding that ‘there is no effect’. Information on possible true effect sizes that are compatible with the data must be obtained from the point estimate, e.g., from a sample average, and from the interval estimate, such as a confidence interval. We review how confusion about interpretation of larger p-values can be traced back to historical disputes among the founders of modern statistics. We further discuss potential arguments against removing significance thresholds, for example that decision rules should rather be more stringent, that sample sizes could decrease, or that p-values should better be completely abandoned. We conclude that whatever method of statistical inference we use, dichotomous threshold thinking must give way to non-automated informed judgment.

show abstract

Section: Introductionmentioning

confidence: 99%

The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research

Amrhein

Korner‐Nievergelt

Roth

2017

PeerJ

266

216

View full text Add to dashboard Cite

show abstract

“…However, does it suffice to settle with, all is well? Obviously the lack of consensus does not help much with moving forward [110,111]. Much of the controversy surrounding statistical significance can be dispelled through a better appreciation of uncertainty and replicability [28].…”

Section: Discussionmentioning

confidence: 99%

Adaptations on the Use of p-Values for Statistical Inference: An Interpretation of Messages from Recent Public Discussions

Verykouki

Nakas

2023

Stats

View full text Add to dashboard Cite

P-values have played a central role in the advancement of research in virtually all scientific fields; however, there has been significant controversy over their use. “The ASA president’s task force statement on statistical significance and replicability” has provided a solid basis for resolving the quarrel, but although the significance part is clearly dealt with, the replicability part raises further discussions. Given the clear statement regarding significance, in this article, we consider the validity of p-value use for statistical inference as de facto. We briefly review the bibliography regarding the relevant controversy in recent years and illustrate how already proposed approaches, or slight adaptations thereof, can be readily implemented to address both significance and reproducibility, adding credibility to empirical study findings. The definitions used for the notions of replicability and reproducibility are also clearly described. We argue that any p-value must be reported along with its corresponding s-value followed by (1−α)% confidence intervals and the rejection replication index.

show abstract

“…While larger sample sizes could compensate, this can be costly: "increasing the size of clinical trials will reduce their feasibility and increase their duration" (Gaudart et al 2014). In Johnson (2014)'s view, this may not necessarily be such a bad thing, pointing to the excess of false positives and the idea that (in the context of clinical trials) "too many ineffective drugs are subjected to phase III testing [...] wast [ing] enormous human and financial resources".…”

Section: Charles Sanders Peirce 1879mentioning

confidence: 99%

The World of Research Has Gone Berserk: Modeling the Consequences of Requiring “Greater Statistical Stringency” for Scientific Publication

Campbell

Gustafson

2019

The American Statistician

View full text Add to dashboard Cite

In response to growing concern about the reliability and reproducibility of published science, researchers have proposed adopting measures of 'greater statistical stringency', including suggestions to require larger sample sizes and to lower the highly criticized 'p < 0.05' significance threshold. While pros and cons are vigorously debated, there has been little to no modeling of how adopting these measures might affect what type of science is published. In this paper, we develop a novel optimality model that, given current incentives to publish, predicts a researcher's most rational use of resources in terms of the number of studies to undertake, the statistical power to devote to each study, and the desirable pre-study odds to pursue. We then develop a methodology that allows one to estimate the reliability of published research by considering a distribution of preferred research strategies. Using this approach, we investigate the merits of adopting measures of 'greater statistical stringency' with the goal of informing the ongoing debate.

show abstract

Reply to Gelman, Gaudart, Pericchi: More reasons to revise standards for statistical evidence

Cited by 4 publications

References 9 publications

The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research

The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research

Adaptations on the Use of p-Values for Statistical Inference: An Interpretation of Messages from Recent Public Discussions

The World of Research Has Gone Berserk: Modeling the Consequences of Requiring “Greater Statistical Stringency” for Scientific Publication

Contact Info

Product

Resources

About