Quantitative Synthesis of Research Evidence: Multilevel Meta-Analysis

Abstract

Multilevel meta-analysis is an innovative synthesis technique used for the quantitative integration of effect size estimates across participants and across studies. The quantitative summary allows for objective, evidence-based, and informed decisions in research, practice, and policy. Based on previous methodological work, the technique results in powerful, unbiased, and precise effect size estimates. However, its use in practice is limited and its full potential is not yet fully understood. This article aims to bring the multilevel meta-analytic model closer to the applied researcher by introducing the technique at a conceptual level and discussing its full potential and relevance to the field. The procedure of multilevel meta-analysis is illustrated using a recent single-case meta-analytic dataset. Software codes, output tables, interpretations, and graphical displays of effect size estimates are given such that the reader can repeat the analysis independently.

Studi Meta-Analisis: Efektivitas Model Project-Based Learning Terhadap Kemampuan Berpikir Kreatif Matematis Siswa

“…Adapun cara menganalisis dan mensintesis secara kuantitatif dari beberapa studi yang dikumpulkan baik dari konferensi, artikel jurnal, skripsi, disertasi dan tesis untuk menyediakan informasi penting bagi para pendidik butuhkan dan pembuat kebijakan (Oh-Young et al, 2020;Moeyaert, 2019;Tamur & Juandi, 2020;Tamur, Juandi, & Kusumah, 2020). Studi meta-analisis merupakan suatu cara yang paling utama dalam menggabungkan beberapa penelitian, memperkuat tingkat validitas hasil penelitian yang ada dengan temuan serupa dan mampu menjelaskan perbedaan tersebut jika terdapat perbedaan hasil (Kot et al, 2018;Siegel et al, 2021;Suparman, Juandi, et al, 2021).…”

Assessing consistency of effects when applying multilevel models to single-case data

“…One the one hand, replication in the SCED context can refer both to repeated demonstrations of a basic effect (e.g., a difference between two adjacent phases) in the same study (Ninci, 2019) and to the replication of effects across studies in relation to the way in which a practice can be established as being "evidence-based" (Jenson et al, 2007;Schlosser, 2009). On the other hand, multilevel models, which are the focus of the current text, have a noteworthy application for meta-analysis (Moeyaert, 2019;Van den Noortgate & Onghena, 2003a, 2003b. In the current text, we here focus on within-study replication and the use of multilevel models as in studies using multiple-baseline designs (Ferron et al, 2009).…”

“…However, there are three reasons why we do not recommend using the statistical significance of the variance as a criterion. First, there are different ways to assess statistically the importance of a random effect: via a Z test under the assumption that the sampling distribution of the variances is normal (Moeyaert, 2019) or comparing the deviance values (−2 times the log likelihood) of the models with and without the random effect via a chisquare test (Hox, 2010). These two tests need not necessary coincide, and both are suspect with small sample sizes, because the variance estimates are biased in such contexts (Ferron et al, 2009).…”

Multilevel Modeling in Single-Case Studies with Count and Proportion Data: A Demonstration and Evaluation

“…In the past decade, the statistical properties of the multilevel meta-analysis methods have been extensively evaluated for continuous outcomes (Moeyaert et al, 2013;Moeyaert, Ugille, et al, 2014;Owens & Ferron, 2012;Ugille et al, 2012). Although there is an ongoing calling for future research to evaluate the performance of Poisson or binomial models and their extensions in meta-analyses of SCEDs (Moeyaert, 2019;Moeyaert et al, 2020;Shadish, 2014a), little is known about the performance of GLMMs and GEE approach in the meta-analysis and there is no consensus about which effect sizes are most appropriate for SCED count and proportion data. The introduced IRR and OR are promising standardized effect sizes to be adopted in the SCED context due to several advantageous features.…”