Why and how we should join the shift from significance testing to estimation

Abstract

A paradigm shift away from null hypothesis significance testing seems in progress. Based on simulations, we illustrate some of the underlying motivations. First, p‐values vary strongly from study to study, hence dichotomous inference using significance thresholds is usually unjustified. Second, ‘statistically significant’ results have overestimated effect sizes, a bias declining with increasing statistical power. Third, ‘statistically non‐significant’ results have underestimated effect sizes, and this bias gets stronger with higher statistical power. Fourth, the tested statistical hypotheses usually lack biological justification and are often uninformative. Despite these problems, a screen of 48 papers from the 2020 volume of the Journal of Evolutionary Biology exemplifies that significance testing is still used almost universally in evolutionary biology. All screened studies tested default null hypotheses of zero effect with the default significance threshold of p = 0.05, none presented a pre‐specified alternative hypothesis, pre‐study power calculation and the probability of ‘false negatives’ (beta error rate). The results sections of the papers presented 49 significance tests on average (median 23, range 0–390). Of 41 studies that contained verbal descriptions of a ‘statistically non‐significant’ result, 26 (63%) falsely claimed the absence of an effect. We conclude that studies in ecology and evolutionary biology are mostly exploratory and descriptive. We should thus shift from claiming to ‘test’ specific hypotheses statistically to describing and discussing many hypotheses (possible true effect sizes) that are most compatible with our data, given our statistical model. We already have the means for doing so, because we routinely present compatibility (‘confidence’) intervals covering these hypotheses.

Quantifying genetic differentiation and population assignment between two contingents of Atlantic mackerel (Scomber scombrus) in the Northwest Atlantic

“…Similarly, Structure has limited discriminatory power in the presence of low genetic differentiation and moderate gene flow (Latch et al 2006;Waples and Gaggiotti 2006), which could explain why clusters were not detected in this study. It can also be argued that the PCA and Structure analyses indicate correctly that there is no distinction between the contingents, but that the observed significant pairwise F ST values represent false-positives due to large data sets (Helyar et al 2011), an argument in line with similar discussions on the importance of focusing on the effect size rather than the significance of P values (Halsey et al 2015;Berner and Amrhein 2022). In this study, effect sizes matched expectations; the largest F ST values (effect size) were observed between the reference sets and the intercountry comparisons while the smallest ones were mainly within Canadian NAFO division comparisons.…”

Durga: An R package for effect size estimation and visualisation

“…P-values cannot be directly compared between studies and often trigger unjustifiable false comparisons (Bernardi et al, 2016;Berner & Amrhein, 2022;Halsey, 2019). A statistical significance indicated by a p-value of less than 0.05 is often erroneously misinterpreted as indicating a meaningful effect size, whereas statistically non-significant results often have an underlying non-zero effect size (Bernardi et al, 2016;Berner & Amrhein, 2022;Halsey, 2019). Furthermore, use of pvalues and NHST effectively asks the binary question, "is there an effect?…”

“…P -values cannot be directly compared between studies and often trigger unjustifiable false comparisons (Bernardi et al, 2016; Berner & Amrhein, 2022; Halsey, 2019). A statistical significance indicated by a p -value of less than 0.05 is often erroneously misinterpreted as indicating a meaningful effect size, whereas statistically non-significant results often have an underlying non-zero effect size (Bernardi et al, 2016; Berner & Amrhein, 2022; Halsey, 2019). Furthermore, use of p -values and NHST effectively asks the binary question, “is there an effect?”, whereas studies in ecology and evolution are typically quantitative and “how large is the effect?” is usually a more important question (Ho et al, 2019; Sullivan & Feinn, 2012).…”

Formal models for the study of the relationship between fluctuating asymmetry and fitness in humans

“…The last component of this framework is the estimation strategy or the process that will be used to learn about the empirical estimand, which includes the estimator and estimate. Among other things, it is sought (see Wasserstein et al, 2019 for an introduction) that rather than estimation strategies that rely on making dichotomous inferences about the presence or absence of the effects of interest (e.g., using null hypothesis significance testing statistical or Bayes factor) and on reporting and interpreting point estimates, estimation strategies focus on estimating the direction and size of these effects, and on embracing uncertainty, for example, by reporting frequentist confidence intervals or their Bayesian counterparts, credible intervals (Berner & Amrhein, 2022;Smith, 2018).…”