Meta-Analyses as a Multi-Level Model

Abstract

Meta-analyses are well known and widely implemented in almost every domain of research in management as well as the social, medical, and behavioral sciences. While this technique is useful for determining validity coefficients (i.e., effect sizes), meta-analyses are predicated on the assumption of independence of primary effect sizes, which might be routinely violated in the organizational sciences. Here, we discuss the implications of violating the independence assumption and demonstrate how meta-analysis could be cast as a multilevel, variance known (Vknown) model to account for such dependency in primary studies’ effect sizes. We illustrate such techniques for meta-analytic data via the HLM 7.0 software as it remains the most widely used multilevel analyses software in management. In so doing, we draw on examples in educational psychology (where such techniques were first developed), organizational sciences, and a Monte Carlo simulation (Appendix). We conclude with a discussion of implications, caveats, and future extensions. Our Appendix details features of a newly developed application that is free (based on R), user-friendly, and provides an alternative to the HLM program.

Multilevel Methods and Statistics: The Next Frontier

“…Two articles provide multilevel extensions for existing statistical analytics approaches. Gooty, Banks, Loignon, Tonidandel, and Williams (2021) describe a multilevel extension for meta-analyses to more fully address lack of independence among underlying studies. In particular, they build on prior work that describes techniques using two-level RCM to account for within-study dependencies (e.g., multiple effects from a single study) by presenting a three-level RCM that also accounts for between-study dependencies (e.g., multiple studies from the same context).…”

“…For multilevel extensions to statistical techniques, Gooty and colleagues (2021) discuss the potential for future research to extend their multilevel meta-analysis technique to account for four levels of analysis and consider cross-classified nesting structures. Additionally, the authors note the need for more consideration with the three-level multilevel meta-analysis approach of accounting for measurement error and range restriction, and utilization of restricted maximum likelihood estimation techniques, which offer benefits for smaller sample sizes.…”

How to conduct a meta-analysis in eight steps: a practical guide

“…Multiple effect sizes from the same study enrich the meta-analytic dataset and allow us to investigate the heterogeneity of the relationship of interest, such as different variable operationalizations (López-López et al 2018;Moeyaert et al 2017). However, including more than one effect size from the same study violates the independency assumption of observations (Cheung 2019;López-López et al 2018), which can lead to biased results and erroneous conclusions (Gooty et al 2021). We follow the recommendation of current best practice guides to take advantage of using all available effect size observations but to carefully consider interdependencies using appropriate methods such as multilevel models, panel regression models, or robust variance estimation (Cheung 2019;Geyer-Klingeberg et al 2020;Gooty et al 2021;López-López et al 2018;Moeyaert et al 2017).…”

Recommendations for Reviewing Meta-Analyses in Organizational Research

“…Meta-analytic regression analysis (MARA) and meta-analytic structural equation modeling (MASEM) represent secondary uses of meta-analytic data (SUMAD; Oh, 2020). These techniques demand mention as they are increasingly used to complement bivariate meta-analyses in organizational research, with articles and shiny apps available for these secondary analyses (Gooty et al, 2019;Yu et al, 2016). SUMAD techniques are hybrids of meta-analysis and more traditional analytic methods (i.e., regression analysis and structural equation modeling), enabling metaanalysts to conduct multivariate analyses.…”