The reign of the p -value is over: what alternative analyses could we employ to fill the power vacuum?

Abstract

The p-value has long been the figurehead of statistical analysis in biology, but its position is under threat. p is now widely recognized as providing quite limited information about our data, and as being easily misinterpreted. Many biologists are aware of p's frailties, but less clear about how they might change the way they analyse their data in response. This article highlights and summarizes four broad statistical approaches that augment or replace the p-value, and that are relatively straightforward to apply. First, you can augment your p-value with information about how confident you are in it, how likely it is that you will get a similar p-value in a replicate study, or the probability that a statistically significant finding is in fact a false positive. Second, you can enhance the information provided by frequentist statistics with a focus on effect sizes and a quantified confidence that those effect sizes are accurate. Third, you can augment or substitute p-values with the Bayes factor to inform on the relative levels of evidence for the null and alternative hypotheses; this approach is particularly appropriate for studies where you wish to keep collecting data until clear evidence for or against your hypothesis has accrued. Finally, specifically where you are using multiple variables to predict an outcome through model building, Akaike information criteria can take the place of the p-value, providing quantified information on what model is best. Hopefully, this quick-and-easy guide to some simple yet powerful statistical options will support biologists in adopting new approaches where they feel that the p-value alone is not doing their data justice.

Be careful with your principal components

“…In the example below, I will use all three ways to show how they can differ in interpretation in each case. The Bayes factor upper bound is calculated as −1/e p ln(p), and gives the support of the alternative hypothesis relative to the null (e.g., Halsey 2019).…”

Reliable confidence intervals for RelTime estimates of evolutionary divergence times

“…We compared RelTime time estimates and CIs with Bayesian time estimates and HPD intervals. We did not test whether the slope between RelTime and Bayesian time estimates was one because p-value will always reject the hypothesis of the slope of one when the data sample size is large, which makes its use less meaningful (Halsey 2019;Wasserstein et al 2019). To compare the performance of our methods and the previous CI calculation methods for RelTime, we re-analyzed all empirical datasets using Mello bounds and the Tamura et al…”

Indices of Effect Existence and Significance in the Bayesian Framework

“…Not only are p-values used to draw inappropriate inferences from noisy data, but even when used properly, effects are drastically overestimated, sometimes even in the wrong direction, when estimation is tied to statistical significance in highly variable data (Gelman, 2018). In response, there is a general agreement that the generalization and utilization of the Bayesian framework is one way of overcoming these issues (Benjamin et al, 2018;Etz & Vandekerckhove, 2016;Halsey, 2019;Marasini, Quatto, & Ripamonti, 2016;Maxwell, Lau, & Howard, 2015;Wagenmakers et al, 2017).…”