Fitting three-level meta-analytic models in R: A step-by-step tutorial

Abstract

Applying a multilevel approach to meta-analysis is a strong method for dealing with dependency of effect sizes. However, this method is relatively unknown among researchers and, to date, has not been widely used in meta-analytic research. Therefore, the purpose of this tutorial was to show how a three-level random effects model can be applied to meta-analytic models in R using the rma.mv function of the metafor package. This application is illustrated by taking the reader through a step-by-step guide to the multilevel analyses comprising the steps of (1) organizing a data file; (2) setting up the R environment; (3) calculating an overall effect; (4) examining heterogeneity of within-study variance and between-study variance; (5) performing categorical and continuous moderator analyses; and (6) examining a multiple moderator model. By example, the authors demonstrate how the multilevel approach can be applied to meta-analytically examining the association between mental health disorders of juveniles and juvenile offender recidivism. In our opinion, the rma.mv function of the metafor package provides an easy and flexible way of applying a multi-level structure to meta-analytic models in R. Further, the multilevel meta-analytic models can be easily extended so that the potential moderating influence of variables can be examined.

The Associations Between Online Media Use and Users’ Perceived Social Resources: A Meta-Analysis

“…In this study, it was necessary to account for the special multilevel data structure (possibly multiple non-independent effect sizes in one individual study), so a three-level random-effects model (REM; Assink & Wibbelink, 2016;Konstantopoulos, 2011) with restricted maximum likelihood and robust variance estimation (RVE) was used (for a detailed introduction and a critical discussion of this method, see Hedges, Tipton, & Johnson, 2010). First, a standard three-level REM was calculated, in which the particular effect sizes were situated on level two, and the primary studies on level three.…”

Executive functions in trauma-exposed youth: a meta-analysis

“…Cohen's d was calculated using reported means and standard deviations, and reported correlations were transformed to Cohen's d. The SPSS syntax for effect size calculation was double-checked by the second author. We used a three-level meta-analytic random effects model as it increases power (Assink & Wibbelink, 2016). It gives us more information because effect sizes are not eliminated or averaged (Assink & Wibbelink, 2016;Cheung, 2014).…”

“…Moderator analyses can explain within-or between-study differences in effect sizes when there is heterogeneity (Borenstein et al, 2010). We used an expert tutorial (Assink & Wibbelink, 2016) for the software R to perform statistical analyses for our three-level meta-analyses with a random model using the Metafor package (Viechtbauer, 2006).…”

Predicting child maltreatment: A meta-analysis of the predictive validity of risk assessment instruments

“…Consequently, we extracted all possible effect sizes from each included primary study, implying that from most primary studies more than one effect size was extracted (see Assink & Wibbelink, 2016 for this procedure). However, a key assumption in traditional metaanalytic approaches is that included effect sizes are independent, so including multiple effect sizes based on the same sample violates this assumption (Lipsey & Wilson, 2001 Weisz et al, 2013), a multilevel random effects model was used for the calculation of combined effect sizes and for the moderator analyses in order to deal with dependency of effect sizes (Hox, 2002;Van den Noortgate & Onghena, 2003).…”

“…We used the R syntax as given by Assink and Wibbelink (2016), so that the three sources of variance as described by for instance Van den Noortgate et al (2013, 2014 were modeled. We applied the Knapp and Hartung (2003) adjustment in estimating coefficients of the multilevel meta-analytic model, meaning that the t-distribution (instead of the z-distribution) was used for testing individual regression coefficients and for calculating the corresponding confidence intervals.…”