Equivalence Testing and the Second Generation P-Value

Abstract

To move beyond the limitations of null-hypothesis tests, statistical approaches have been developed where the observed data are compared against a range of values that are equivalent to the absence of a meaningful effect. Specifying a range of values around zero allows researchers to statistically reject the presence of effects large enough to matter, and prevents practically insignificant effects from being interpreted as a statistically significant difference. We compare the behavior of the recently proposed second generation p-value (Blume, D'Agostino McGowan, Dupont, & Greevy, 2018) with the more established Two One-Sided Tests (TOST) equivalence testing procedure (Schuirmann, 1987). We show that the two approaches yield almost identical results under optimal conditions. Under suboptimal conditions (e.g., when the confidence interval is wider than the equivalence range, or when confidence intervals are asymmetric) the second generation p-value becomes difficult to interpret. The second generation p-value is interpretable in a dichotomous manner (i.e., when the SGPV equals 0 or 1 because the confidence intervals lies completely within or outside of the equivalence range), but this dichotomous interpretation does not require calculations. We conclude that equivalence tests yield more consistent p-values, distinguish between datasets that yield the same second generation p-value, and allow for easier control of Type I and Type II error rates.To test predictions researchers predominantly rely on nullhypothesis tests. This statistical approach can be used to examine whether observed data are sufficiently surprising under the null hypothesis to reject an effect that equals exactly zero. Null-hypothesis tests have an important limitation, in that this procedure can only reject the hypothesis that there is no effect, while scientists should also be able to provide statistical support for equivalence. When testing for equivalence researchers aim to examine whether an observed effect is too small to be considered meaningful, and therefore is practically equivalent to zero. By specifying a range around the null hypothesis of values that are deemed practically equivalent to the absence of an effect (i.e., 0 ± 0.3) the observed data can be compared against an equivalence range and researchers can test if a meaningful effect is absent ; Westlake, 1972).Second generation p-values (SGPV) were recently proposed as a statistic that represents "the proportion of data-supported hypotheses that are also null hypotheses" (Blume et al., 2018). The researcher specifies an equivalence range around a null hypothesis of values that are considered practically equivalent to the null hypothesis. The SGPV measures the degree to which a set of data-supported parameter values falls within the interval null hypothesis. If the estimation interval falls completely within the equivalence range, the SGPV is 1. If the confidence interval falls completely outside of the equivalence range, the SGPV is 0. Otherwise the SGPV is a value between 0 and 1...

The Practical Alternative to the p Value Is the Correctly Used p Value

“…In some research articles in sports science, p values were replaced by magnitude-based inferences ( Batterham & Hopkins, 2006 ), which were recently strongly criticized because of their high error rates ( Sainani, 2018 ). Recently proposed “second-generation p values” ( Blume, D’Agostino McGowan, Dupont, & Greevy, 2018 ) turned out to be highly similar to, but less informative than, equivalence tests ( Lakens & Delacre, 2020 ). Training researchers how to use existing frequentist and Bayesian approaches to estimation and hypothesis testing well (which means with care and while acknowledging the limitations of each approach) might be a more fruitful approach for improving statistical inferences than developing novel statistical approaches.…”

Basic statistical considerations for physiology: The journalTemperaturetoolbox

“…In these cases, traditional hypothesis tests are not appropriate because a nonsignificant p-value does not provide evidence that there is zero or no effect at all [49,56]. Researchers should avoid using "magnitude based inference" [57,58] or "second generation p-values" [59] as hypothesis tests for equivalence, though they may be useful as descriptive statistics of the confidence interval. The primary problem with using these statistics as hypothesis tests is the higher false positive risk and dependence upon the sample size [57].…”

New relevance and significance measures to replace p-values

“…Then, P ζ has a maximum value of 1/2, which is a deliberate consequence of the definition, as this value does not suggest a “proof” of H 0 . For a comparison of the SGPV with TOST, see [ 12 ].…”

“…4.2.6 Estimation. The effects η J and ϑ pred,J are estimated by plugging in estimates for the parameters,b J ,ŝ ¼ŝ f andŝ r , into (12) or (13). Using the first option shows how to obtain a confidence interval.…”