Revised standards for statistical evidence

Abstract

Recent advances in Bayesian hypothesis testing have led to the development of uniformly most powerful Bayesian tests, which represent an objective, default class of Bayesian hypothesis tests that have the same rejection regions as classical significance tests. Based on the correspondence between these two classes of tests, it is possible to equate the size of classical hypothesis tests with evidence thresholds in Bayesian tests, and to equate P values with Bayes factors. An examination of these connections suggest that recent concerns over the lack of reproducibility of scientific studies can be attributed largely to the conduct of significance tests at unjustifiably high levels of significance. To correct this problem, evidence thresholds required for the declaration of a significant finding should be increased to 25-50:1, and to 100-200:1 for the declaration of a highly significant finding. In terms of classical hypothesis tests, these evidence standards mandate the conduct of tests at the 0.005 or 0.001 level of significance. R eproducibility of scientific research is critical to the scientific endeavor, so the apparent lack of reproducibility threatens the credibility of the scientific enterprise (e.g., refs. 1 and 2). Unfortunately, concern over the nonreproducibility of scientific studies has become so pervasive that a Web site, Retraction Watch, has been established to monitor the large number of retracted papers, and methodology for detecting flawed studies has developed nearly into a scientific discipline of its own (e.g., refs. 3-9).Nonreproducibility in scientific studies can be attributed to a number of factors, including poor research designs, flawed statistical analyses, and scientific misconduct. The focus of this article, however, is the resolution of that component of the problem that can be attributed simply to the routine use of widely accepted statistical testing procedures.Claims of novel research findings are generally based on the outcomes of statistical hypothesis tests, which are normally conducted under one of two statistical paradigms. Most commonly, hypothesis tests are performed under the classical, or frequentist, paradigm. In this approach, a "significant" finding is declared when the value of a test statistic exceeds a specified threshold. Values of the test statistic above this threshold define the test's rejection region. The significance level α of the test is defined to be the maximum probability that the test statistic falls into the rejection region when the null hypothesis-representing standard theory-is true. By long-standing convention (10), a value of α = 0.05 defines a significant finding. The P value from a classical test is the maximum probability of observing a test statistic as extreme, or more extreme, than the value that was actually observed, given that the null hypothesis is true.The second approach for performing hypothesis tests follows from the Bayesian paradigm and focuses on the calculation of the posterior odds that the alternative hypotheses is true, give...

Performance and metabolite profile of dairy cows fed tropical grasses and concentrates containing crude protein with low or high degradability

“…However, as we did not evaluate the body composition of cows and a Latin square as the experimental design was adopted, we cannot conclusively affirm what may have caused the significant difference in creatinine excretion observed for the CS × Others contrast. Nonetheless, the P-values obtained for this variable using the applied frequentist statistical analysis were, from a Bayesian point of view, close to false positive detection levels (Johnson, 2013).…”

Expressive language and prosocial behaviour in early childhood: Longitudinal associations in the UK Millennium Cohort Study

“…Due to the large study sample we adopted a more stringent statistical threshold: alpha = .001 instead of .050 (see Johnson, 2013). Missing data, regarding covariates from non-response, were treated using Full Information Maximum Likelihood (FIML), which has been found to be an unbiased and advantageous approach in handling missing data (Enders & Bandalos, 2001).…”

Null Hypothesis Significance Testing: a short tutorial

“…The significance level α is defined to be the maximum probability that a test statistic falls into 73 the rejection region when the null hypothesis is true (Johnson, 2013). Therefore, one can only 74 reject the null hypothesis if the test statistics falls into the critical region(s), or fail to reject 75 this hypothesis.…”