Comparative Probability Metrics: Using Posterior Probabilities to Account for Practical Equivalence in A/B tests

Bayesian Mendelian randomization testing of interval causal null hypotheses: ternary decision rules and loss function calibration

“…The first is a Bayesian approach to inference, whereby evidence in favour and against the null can be separately assessed. The second is replacing the point null hypothesis with a region of practical equivalence (ROPE) consisting of values of negligible magnitude for the effect of interest (Kruschke [1]; Stanton [7]; Kelter [8]; Kelter [9]; Liao et al [10]; Maximilian et al [11]; Stevens and Hagar [12]), and exploiting the ability of Bayesian analysis to quantify the evidence of the effect falling inside/outside the region. Third: we move away from a binary decision, where the outcome of the hypothesis test is either accept or reject, in favour of a ternary logic that accommodates a third, "uncertain", outcome of the test.…”