Four reasons to prefer Bayesian analyses over significance testing

Abstract

Inference using significance testing and Bayes factors is compared and contrasted in five case studies based on real research. The first study illustrates that the methods will often agree, both in motivating researchers to conclude that H1 is supported better than H0, and the other way round, that H0 is better supported than H1. The next four, however, show that the methods will also often disagree. In these cases, the aim of the paper will be to motivate the sensible evidential conclusion, and then see which approach matches those intuitions. Specifically, it is shown that a high-powered non-significant result is consistent with no evidence for H0 over H1 worth mentioning, which a Bayes factor can show, and, conversely, that a low-powered non-significant result is consistent with substantial evidence for H0 over H1, again indicated by Bayesian analyses. The fourth study illustrates that a high-powered significant result may not amount to any evidence for H1 over H0, matching the Bayesian conclusion. Finally, the fifth study illustrates that different theories can be evidentially supported to different degrees by the same data; a fact that P-values cannot reflect but Bayes factors can. It is argued that appropriate conclusions match the Bayesian inferences, but not those based on significance testing, where they disagree.

Multilevel convergence of interoceptive impairments in hypertension: New evidence of disrupted body–brain interactions

“…Finally, no significant between‐group differences emerged in the VBM analysis ( p ‐value < .001 uncorrected, spatial threshold = 50 voxels; see Supporting Information, Figure 1), in FA or MD from our DTI analyses ( p < .001 uncorrected, spatial threshold = 50 voxels; see Supporting Information, Figure 2), or in the functional connectivity comparison based on the voxel‐wise correlation of the short gyri of the left insula and the rest of the brain ( p value < .001 uncorrected, spatial threshold = 50 voxels; see Supporting Information, Figure 3). In all these neuroimaging analyses, the Bayes factor was under 1 and, in most cases, between 0 and 0.33, supporting the null hypothesis (Dienes, ; Dienes & Mclatchie, ), namely, that there were no significant differences between groups (Supporting Information, Tables 3 and 4). In sum, convergent behavioral, VBM, DTI, and functional connectivity results revealed preserved cognitive and brain profiles in the patients.…”

“…Also, to confirm the robustness of our negative results in HR, HRV, and blood pressure during the HBD task, we established the probability with which the null hypothesis can be accepted or rejected via a Bayes analysis on JASP Statistical Software (https://jasp-stats.org/, JASP Team (2017) (Version 0.8.2); (Rouder, Speckman, Sun, Morey, & Iverson, ). We performed Bayes tests for all results yielding non‐significant differences (Dienes, ; Dienes, Coulton, & Heather, ; Dienes & Mclatchie, ). A Bayes factor of 0 to 0.33 strongly supports the null hypothesis, whereas a Bayes factor between 0.33 and 3 indicates insufficient evidence to do so.…”

“…In addition, we performed Bayes tests for all results yielding nonsignificant differences (Dienes, ; Dienes et al, ; Dienes & Mclatchie, ), to establish the probability with which the null hypothesis can be accepted or rejected. This calculation was performed for the structural (DTI, VBM) and functional connectivity differences between groups.…”

Stereotype threat may not impact women's inhibitory control or mathematical performance: Providing support for the null hypothesis

“…In line with recommendations by Dienes and McLatchie (2018), we then calculated Bayes factors (B) for all 1 degree of freedom effects (see Martin, Sackur, Anll o, Naish, & Dienes, 2016 for similar analyses). The results were analysed using both Null Hypothesis Significance Testing (NHST) and Bayesian analyses.…”

The Bayesian Revolution in Second Language Research: An Applied Approach

“…Recent years have seen repeated calls to reform the conventional data analysis practices in the social and behavioral sciences (e.g., Dienes & Mclatchie, ; Etz & Vandekerckhove, ; Kruschke & Liddell, ; Norouzian & Plonsky, , in press). Most prominent among these calls, however, has been one to shift emphasis away from frequentist methods to Bayesian methods.…”