Brief Research Report: Bayesian Versus REML Estimations With Noninformative Priors in Multilevel Single-Case Data

Abstract

Recently, researchers have used multilevel models for assessing intervention effects in singlecase studies, which are based on the replication of interrupted time-series designs across a small number of cases. Researchers estimating these multilevel models have primarily relied on restricted maximum likelihood (REML) techniques, but Bayesian approaches have also been suggested. The purpose of this Monte Carlo simulation study was to examine the impact of estimation method (REML versus Bayesian with noninformative priors) on the estimation of treatment effects (relative bias, root mean square error) and on the inferences about those effects (interval coverage) for autocorrelated single-case data. Simulated conditions varied with regard to the number of participants, series length, and the distribution of the variance within and across cases. REML and Bayesian estimation led to unbiased estimates of the fixed effects, but differentially impacted the inferences about the fixed effects and the estimates of the variances.Implications for applied single-case researchers and methodologists are discussed.

Modeling heterogeneity of the level-1 error covariance matrix in multilevel models for single-case data

“…Similarly, Gelman (2006) found that when the uniform distribution (noninformative prior) is constructed for the standard deviation (SD) unit rather than variance unit of level-2 error, the uniform distribution generally performs well, as long as J ≥ 3 which is required to ensure a proper posterior density. Baek et al (2019) examined the impact of estimation methods in multilevel SCED using noninformative priors for variance parameters, and their study yielded similar results with the previous study that constructed weakly informative priors. Based on these suggestions, noninformative prior distributions for the SD of the level-2 errors (σ u 0j , σ u 1j , σ u 2j , σ u 3j ) and the level-1 errors (σ e ) were assigned to be the uniform distribution (lower limit = 0 and upper limit = 100) in this study.…”

“…Thus, Bayesian estimation also offers practical advantages be-cause it takes into account the uncertainty of estimating both fixed effects and variance components (Gelman, Carlin, Stern, & Rubin, 2004). In addition to the computational and practical benefits of using Bayesian estimation, recent studies have indicated that the Bayesian approach has potential benefits in estimating effect sizes, analyzing nonlinear data, and estimating autocorrelation (Baek et al, 2019;Moeyaert, Rindskopf, Onghena, &Van den Noortgate, 2017;Rindskopf, 2014a;Rindskopf, 2014b;Shadish et al, 2013;Swaminathan, Rogers, & Horner, 2014). Baek et al (2019) examined the impact of REML and Bayesian estimation on average treatment effect inferences of SCED studies using multilevel modeling.…”

“…Various noninformative and weakly informative prior distributions have been suggested for the variance parameters in multilevel models, including uniform, inverse-gamma family, inverse-Wishart, half-Cauchy, and half-t distributions (Baek et al, 2019;Berger & Strawderman, 1996;Daniels & Kass, 1999;Gelman, 2006;Moeyaert et al, 2017). Moeyaert et al (2017) constructed various weakly informative priors for variance components for SCED data, and the results found biased and less precise variance estimates when the number of cases is small (J = 3).…”

Effects of Using Virtual Manipulatives for Students With Disabilities: Three-Level Multilevel Modeling for Single-Case Data

“…Using three-level multilevel modeling, we examined the immediate effects and trends during the intervention as well as the moderation effects of case-level (student characteristics) and study-level (intervention features) variables. Multilevel models for individual case data allowed for flexibility and addressed methodological considerations within SCD studies, such as changes in levels and trends between baseline and intervention phases (Baek et al, 2020), variability in intervention effects through moderators at both the case and study levels (Moeyaert et al, 2020), and within-group errors that can be autocorrelated (Petit-Bois et al, 2016). Three primary research questions guided the current study:…”

Assessing consistency of effects when applying multilevel models to single-case data

“…Regarding the proposal of quantifying the percentage of random effect confidence intervals that include 0, it should be noted that this percentage is not expected to approximate any theoretically desirable quantity. In that sense, we are not quantifying how many of the confidence intervals in different samples or replications are including a population parameter, which would be equivalent to studying the coverage of a confidence interval (e.g., Baek et al, 2019;Ferron et al, 2009;Moeyaert et al, 2017), expected to be .95 for a 95% confidence.…”