Bayesian Repeated-Measures Analysis of Variance: An Updated Methodology Implemented in JASP

Abstract

Analysis of variance (ANOVA) is widely used to assess the influence of one or more experimental (or quasi-experimental) manipulations on a continuous outcome. Traditionally, ANOVA is carried out in a frequentist manner using p values, but a Bayesian alternative has been proposed. Assuming that the proposed Bayesian ANOVA is closely modeled after its frequentist counterpart, one may be surprised to find that the two can yield very different conclusions when the design involves multiple repeated-measures factors. We illustrate such a discrepancy with a real data set from a two-factorial within-subject experiment. For this data set, the results of a frequentist and Bayesian ANOVA are in a disagreement about which main effect accounts for the variance in the data. The reason for this disagreement is that frequentist and the proposed Bayesian ANOVA use different model specifications. As currently implemented, the proposed Bayesian ANOVA assumes that there are no individual differences in the magnitude of effects. We suspect that this assumption is neither obvious to nor desired by most analysts because it is untenable in most applications. We argue here that the Bayesian ANOVA should be revised to allow for individual differences. As a default, we suggest the standard frequentist model specification but discuss a recently proposed alternative and provide guidance on how to choose the appropriate model specification. We end by discussing the implications of the revised model specification for previously published results of Bayesian ANOVAs.

Data aggregation can lead to biased inferences in Bayesian linear mixed models and Bayesian analysis of variance.

“…We used SBC to demonstrate that analyses based on aggregate data can lead to biases in the resulting null hypothesis tests, and should therefore be avoided. We demonstrated that biases in inferences do not only occur in frequentist approaches, where these are well-known and routinely corrected using frequentist tools, but that they also apply in Bayesian data analysis using Bayes factors, where they are often neglected (van den Bergh et al, 2023).…”

“…For such aggregated data, it is not possible to estimate all random effect variances for the different effects (contrasts). Instead, only some of the random effects are informed by the data (van den Bergh et al, 2023). For example, in a one-factor design, only random intercepts are informed, but not random slopes; in a two-factor design, only random slopes for the main effects are informed by the data, but not random slopes for the interaction.…”

“…When analyzing a 2 × 2 design with aggregated data using LMMs, there are different possible approaches to data analysis. First, we can simply ignore random slopes as we did in the examples with only one factor level—this is indeed the default setting in Bayesian repeated measures ANOVA in the R-package , and it used to be the default approach in JASP (van den Bergh et al, 2023). However, in a 2 × 2 design, there are two data points available for each subject and each factor level.…”

“…(These estimates based on two data points may not be very stable.) The interaction term, however, is confounded with the residual noise again, and random slopes are not fully informed by the data (and cannot be estimated in frequentist LMMs; van den Bergh et al, 2023). We expected that although some between-subject variance can be captured in this analysis, two data points are not enough to capture this source of variance fully (i.e., of all random slopes), and we therefore expected that the biases should still exist when the interaction term has a large variance.…”

Separating facilitation and interference in backward crosstalk.

“…The scaling of priors was set to default, and BFs were calculated in analogy to the Type III tests in classical frequentist ANOVA. Note that there has been a recent debate about the shortcomings of Bayesian ANOVA (when implemented as a mixed model) with default settings (Singmann et al, 2023; van den Bergh et al, 2023). We thus report the corresponding BFs for consistency, but remain a main focus on the classical frequentist ANOVA.…”