The application of meta-analytic (multi-level) models with multiple random effects: A systematic review

Abstract

In meta-analysis, study participants are nested within studies, leading to a multilevel data structure. The traditional random effects model can be considered as a model with a random study effect, but additional random effects can be added in order to account for dependent effects sizes within or across studies. The goal of this systematic review is three-fold. First, we will describe how multilevel models with multiple random effects (i.e., hierarchical three-, four-, five-level models and cross-classified random effects models) are applied in meta-analysis. Second, we will illustrate how in some specific three-level meta-analyses, a more sophisticated model could have been used to deal with additional dependencies in the data. Third and last, we will describe the distribution of the characteristics of multilevel meta-analyses (e.g., distribution of the number of outcomes across studies or which dependencies are typically modeled) so that future simulation studies can simulate more realistic conditions. Results showed that four-or five-level or cross-classified random effects models are not often used although they might account better for the meta-analytic data structure of the analyzed datasets. Also, we found that the simulation studies done on multilevel metaanalysis with multiple random factors could have used more realistic simulation factor conditions. The implications of these results are discussed, and further suggestions are given.

Mental health problems in the general population during and after the first lockdown phase due to the SARS-Cov-2 pandemic: rapid review of multi-wave studies

“…Additionally, we found many studies that used more than one indicator answered by the same respondents. This methodological challenge is known from previous reviews on prevalence changes of mental illness and needs to be accounted for as the effect sizes are dependent (Richter et al ., 2019 ; Fernandez-Castilla et al ., 2020 ). In terms of psychopathology, we found several mental health problems that were addressed: anxiety, depression and distress among them prominently.…”

To What Extent Does the Involvement Load Hypothesis Predict Incidental L2 Vocabulary Learning? A Meta‐Analysis

“…The rest (11 studies, 47.8%) did not report how they determined the proficiency level. 8 The three-level meta-regression models used in the current study can be seen as an extension of the traditional random-effects model (Hedges & Olkin, 1985), which is equivalent to a two-level meta-regression model (e.g., Fernández-Castilla et al, 2020). 9 One might wonder whether the lack of contribution of search to learning could have been due to the fact that the majority of the studies included learning conditions without search, for example, the form and meaning of the target words were provided as marginal glosses in the task (see Appendix S6 in the Supporting Information online).…”

“…88The three‐level meta‐regression models used in the current study can be seen as an extension of the traditional random‐effects model (Hedges & Olkin, 1985), which is equivalent to a two‐level meta‐regression model (e.g., Fernández‐Castilla et al., 2020). …”

Prevalence of sending, receiving and forwarding sexts among youths: A three-level meta-analysis

“…By ignoring the dependence in effect sizes, the two-level model can result in standard errors that are too small, and therefore in largely deflated coverage proportions of confidence intervals [28]. Furthermore, the application of a three-level meta-analysis is especially appropriate when the outcomes of interest vary in measurement form across studies [29].…”