Indices of Effect Existence and Significance in the Bayesian Framework

Abstract

Turmoil has engulfed psychological science. Causes and consequences of the reproducibility crisis are in dispute. With the hope of addressing some of its aspects, Bayesian methods are gaining increasing attention in psychological science. Some of their advantages, as opposed to the frequentist framework, are the ability to describe parameters in probabilistic terms and explicitly incorporate prior knowledge about them into the model. These issues are crucial in particular regarding the current debate about statistical significance. Bayesian methods are not necessarily the only remedy against incorrect interpretations or wrong conclusions, but there is an increasing agreement that they are one of the keys to avoid such fallacies. Nevertheless, its flexible nature is its power and weakness, for there is no agreement about what indices of "significance" should be computed or reported. This lack of a consensual index or guidelines, such as the frequentist p-value, further contributes to the unnecessary opacity that many nonfamiliar readers perceive in Bayesian statistics. Thus, this study describes and compares several Bayesian indices, provide intuitive visual representation of their "behavior" in relationship with common sources of variance such as sample size, magnitude of effects and also frequentist significance. The results contribute to the development of an intuitive understanding of the values that researchers report, allowing to draw sensible recommendations for Bayesian statistics description, critical for the standardization of scientific reporting.

Analysis of Bayesian posterior significance and effect size indices for the two-sample t-test to support reproducible medical research

“…The probability of direction is detailed in Makowski et al [13] and varies between 50% and 100%. It is defined as the proportion of the posterior distribution of the parameter that is of the median's sign.…”

“…The MAP-based p-value was proposed by Mills [14] (see also Makowski et al [13]), and can be related to the odds that a parameter has against the null hypothesis: It is defined as the ratio of the posterior density at the null value and the value of the posterior density at the maximum a posteriori (MAP) value, which is the equivalent of the mode for continuous probability distributions.…”

“…Makowski et al [13] also proposed the Bayes factor versus ROPE index, which does not compare the point null hypothesis H 0 : δ = 0 against an alternative H 1 : δ = 0 as the normal BF, but used a null H 0 : δ ∈[ −0.1, 0.1] which is given by the ROPE and then tests against the alternative H 1 : δ / ∈[ −0.1, 0.1] which is the complement to the ROPE. While this approach is highly similar to the traditional ROPE and shows similar behaviour indeed [13], it will not be used here.…”

“…There are measures which state the significance of an effect, and measures which also gauge the size of it. Among them is the Bayes factor introduced by Jeffreys [11], the region of practical equivalence (ROPE) championed by Kruschke [12], the probability of direction (PD) as detailed in Makowski et al [13], the MAP-based p-value proposed by Mills [14], and the Full Bayesian Significance Test (FBST) featuring the e-value, which was introduced by Pereira, Stern and Wechsler [15,16]. The appropriateness of these indices is still debated in the lit-erature, which makes it challenging to choose among the available indices because by now there is no explicit agreement on which index researchers should use to report the results of a Bayesian analysis [10,[17][18][19].…”

Large prospective losses lead to sub-optimal sensorimotor decisions in humans

“…Probability of Direction (pd) (Makowski et al, 2019). Figure 2a shows posterior 169 probability densities over sizes of our effects of interest, Figure 2b shows predicted 170 marginal effects of interest, and Figure 2c shows posterior predictive checks against 171 observed means and standard deviations.…”