Valid P -Values Behave Exactly as They Should: Some Misleading Criticisms of P -Values and Their Resolution With S -Values

Abstract

The present note explores sources of misplaced criticisms of P-values, such as conflicting definitions of "significance levels" and "P-values" in authoritative sources, and the consequent misinterpretation of P-values as error probabilities. It then discusses several properties of P-values that have been presented as fatal flaws: That P-values exhibit extreme variation across samples (and thus are "unreliable"), confound effect size with sample size, are sensitive to sample size, and depend on investigator sampling intentions. These properties are often criticized from a likelihood or Bayesian framework, yet they are exactly the properties P-values should exhibit when they are constructed and interpreted correctly within their originating framework. Other common criticisms are that P-values force users to focus on irrelevant hypotheses and overstate evidence against those hypotheses. These problems are not however properties of P-values but are faults of researchers who focus on null hypotheses and overstate evidence based on misperceptions that p = 0.05 represents enough evidence to reject hypotheses. Those problems are easily seen without use of Bayesian concepts by translating the observed P-value p into the Shannon information (S-value or surprisal) −log 2 (p).

Insight into biogeochemical models from Scale Transition Theory: A dimensionless, scale-free approach

“…based on leave-one-out cross validation), in addition to metrics of model misfit or discrepancy (e.g. Sander Greenland's S-value, Greenland (2019)).…”

A Frequentist Alternative to Significance Testing, p-Values, and Confidence Intervals

“…And yet, there is a potential way out. Greenland (2019) suggested performing a logarithmic transformation of p-values: (−1)· log 2 (p). The logarithmic transformation causes p-values to be expressed as the number of bits of information against the model.…”

Inferential statistics as descriptive statistics: there is no replication crisis if we don't expect replication

“…But the P-value itself is not supposed to be "reliable" in the sense of staying put (Greenland 2018a). Its fickleness indicates variation in the data from sample to sample.…”

“…Instead of talking about hypothetical coverage of the true value by such intervals, which will fail under various assumption violations, we can think of the confidence interval as a "compatibility interval" (Greenland 2018a,b), showing effect sizes most compatible with the data according to their P-values, under the model used to compute the interval. Likewise, we can think of a posterior probability interval, or Bayesian "credible interval," as a compatibility interval showing effect sizes most compatible with the data, under the model and prior distribution used to compute the interval (Greenland 2018a). Again, whether such intervals include or exclude zero should play no role in their interpretation, because even with only random variation the intervals from different studies can easily be very different (Cumming 2014).…”