To curb research misreporting, replace significance and confidence by compatibility

Divergence versus decisionP‐values: A distinction worth making in theory and keeping in practice: Or, how divergenceP‐values measure evidence even when decisionP‐values do not

“…As the divergence increases, sc increases to reflect surprise at the data given the model, or conflict with the target model (Bayarri & Berger, 1999;Greenland, 2019a;Cole et al 2021;Gibson, 2021;Amrhein & Greenland, 2022;Greenland et al, 2022). The base of the log, c, can be seen as a scale factor; c=10 is commonly used but has no theoretical justification.…”

“…Assuming D A is a real scalar random variable such as a χ 2 statistic, the observed upper‐tail P ‐value derived from the realization D A = d is p d ; A = g y ; A ({ D A ≥ d }) = ∫ u ≥ d g y ; A ( u ) du , which is the ordinal location or rank of d in the reference distribution. It is interpreted as a unit‐scaled index of consonance, consistency, or compatibility of y with F A (Amrhein & Greenland, 2022; Amrhein et al, 2019; Bayarri & Berger, 2000; Box, 1980, p. 347; Cox, 1977; Cox & Hinkley, 1974, p. 66; Folks, 1981; Greenland, 2019a; Greenland et al, 2022, 2023; Kempthorne & Folks, 1971; Rafi & Greenland, 2020; Robins et al, 2000). In a complementary fashion, 1 − p d ; A is a unit‐scaled index of incompatibility or inconsistency which summarizes the divergence of y from F A as measured by d ( y ; F A ), although a more equal‐interval measurement of incompatibility or refutational evidence is provided by the S ‐value −log c ( p d ; A ) where c is the log base (Greenland, 2019a, 2021a; Greenland et al, 2022, 2023; Kempthorne, 1990; Rafi & Greenland, 2020).…”

“…It is interpreted as a unit‐scaled index of consonance, consistency, or compatibility of y with F A (Amrhein & Greenland, 2022; Amrhein et al, 2019; Bayarri & Berger, 2000; Box, 1980, p. 347; Cox, 1977; Cox & Hinkley, 1974, p. 66; Folks, 1981; Greenland, 2019a; Greenland et al, 2022, 2023; Kempthorne & Folks, 1971; Rafi & Greenland, 2020; Robins et al, 2000). In a complementary fashion, 1 − p d ; A is a unit‐scaled index of incompatibility or inconsistency which summarizes the divergence of y from F A as measured by d ( y ; F A ), although a more equal‐interval measurement of incompatibility or refutational evidence is provided by the S ‐value −log c ( p d ; A ) where c is the log base (Greenland, 2019a, 2021a; Greenland et al, 2022, 2023; Kempthorne, 1990; Rafi & Greenland, 2020). These definitions and interpretations are sometimes called Fisherian or neo‐Fisherian, although they can be seen in writings before Fisher's, such as Pearson (1900), and could be called evidential or reference frequentist.…”

“…Other terms for the compatibility concept include conformity, concordance, and accord (Rice, personal communication). Its opposite may thus be termed incompatibility or discordance, which can be measured by 1 − p , although again there are cognitive and abstract arguments for instead employing the surprisal measure or S ‐value s = −log c ( p ) for incompatibility (Amrhein & Greenland, 2022; Greenland, 2019a; Greenland, 2021a; Greenland et al, 2022, 2023; Rafi & Greenland, 2020).…”

“…The most common example of such a transform is the negative base‐ c logarithmic transform to a surprisal or S ‐value s c = log c (1/ p ) = −log c ( p ) = −ln( p )/ln( c ); s c = 0 then says the observed discrepancy statistic or divergence measure from which the P ‐value was computed was completely unsurprising given the target model, or that it does not conflict with or contains no information refuting the model. As the divergence increases, s c increases to reflect surprise at the data given the model, or conflict with the target model (Amrhein & Greenland, 2022; Bayarri & Berger, 1999; Cole et al, 2021; Gibson, 2021; Greenland, 2019a; Greenland et al, 2022; Rafi & Greenland, 2020). The base of the log, c , can be seen as a scale factor; c = 10 is commonly used but has no theoretical justification.…”