Redefine statistical significance

Abstract

We propose to change the default P-value threshold for statistical significance from 0.05 to 0.005 for claims of new discoveries. T he lack of reproducibility of scientific studies has caused growing concern over the credibility of claims of new discoveries based on 'statistically significant' findings. There has been much progress toward documenting and addressing several causes of this lack of reproducibility (for example, multiple testing, P-hacking, publication bias and under-powered studies). However, we believe that a leading cause of non-reproducibility has not yet been adequately addressed: statistical standards of evidence for claiming new discoveries in many fields of science are simply too low. Associating statistically significant findings with P < 0.05 results in a high rate of false positives even in the absence of other experimental, procedural and reporting problems.For fields where the threshold for defining statistical significance for new discoveries is P < 0.05, we propose a change to P < 0.005. This simple step would immediately improve the reproducibility of scientific research in many fields. Results that would currently be called significant but do not meet the new threshold should instead be called suggestive. While statisticians have known the relative weakness of using P ≈ 0.05 as a threshold for discovery and the proposal to lower it to 0.005 is not new 1,2 , a critical mass of researchers now endorse this change.We restrict our recommendation to claims of discovery of new effects. We do not address the appropriate threshold for confirmatory or contradictory replications of existing claims. We also do not advocate changes to discovery thresholds in fields that have already adopted more stringent standards (for example, genomics and high-energy physics research; see the 'Potential objections' section below).We also restrict our recommendation to studies that conduct null hypothesis significance tests. We have diverse views about how best to improve reproducibility, and many of us believe that other ways of summarizing the data, such as Bayes factors or other posterior summaries based on clearly articulated model assumptions, are preferable to P values. However, changing the P value threshold is simple, aligns with the training undertaken by many researchers, and might quickly achieve broad acceptance.

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

“…As seen in suggested that the p-value threshold for statistically significant findings should be lowered from 0.05 to 0.005 for new discoveries 30 . In a replication context it would be relevant to apply this stricter threshold to meta-analytic results.…”

Four simple ways to increase power without increasing the sample size

“…[8][9][10][11][12] If journals start requiring a lower threshold for significance, the power of all experiments will be further reduced, exacerbating the above problems. By way of example, suppose we are conducting a two group experiment with independent samples in each group.…”

Why do we need to employ Bayesian statistics and how can we employ it in studies of moral education?: With practical guidelines to use JASP for educators and researchers

“…From the perspective of frequentists, Pvalues only provide information pertaining to whether a null hypothesis about the extremity of an observed distribution can be rejected; they do not say anything about whether and how strongly evidence found from a specific study supports a hypothesis . Furthermore, as the current debates indicated, conventional P-value thresholds widely used in the field, particularly, p < .05, could only support very week or even could not support RUNNING HEAD: UTILIZING BAYESIAN STATISTICS 23 the presence of positive evidence (Benjamin et al, 2018). Instead, BFs show us the strength of evidence; directly BF thresholds used in the field can also be considered as better thresholds to make practical decisions about accepting a specific hypothesis based on evidence (Kass & Raftery, 1995).…”

“…Although recent debates about the frequentist perspective in the field of quantitative methods have intensified concerns regarding how to collect and test data properly (Benjamin et al, 2018), the majority of studies in the fields related to moral education have tend to use such a perspective. We have been used to employing the methodology of frequentist, such as P-values, in empirical studies of moral education.…”