Meta-analysis of second language research with complex research designs

Abstract

Meta-analyses play an instrumental role in informing second language (L2) theory and practice. However, current (i.e., classic) approaches to meta-analysis are limited in their ability to do so because they often fail to capture the complexity inherent in primary studies’ research designs. As we argue in this article, when complex L2 studies are represented by simplistic meta-analyses, the latter cannot reach its potential to contribute to the development of cumulative knowledge. To mitigate this issue, we first discuss the fundamental problems of the classic approaches to meta-analysis of complex L2 research. Second, we introduce an alternative meta-analytic framework that will address those problems. Third, we apply the meta-analytic framework discussed to a complex L2 domain. Fourth, we offer free software to facilitate the conduct of the alternative meta-analytic approach described. Finally, we discuss the implications of this alternative framework for making evidence-based recommendations to the relevant stakeholders.

Complex Adaptive Interventions: The Challenge Ahead for Instructed Second Language Acquisition Research

“…While funding agencies and stakeholders in scholarly publishing make treatment fidelity a standard part of the conduct and evaluation of educational intervention research, this position is contested by the increasingly complex designs and interventions used in practice, which render effects heterogenous by default (see, e.g., Bryan et al, 2021;Norouzian & Bui, 2023). The reality is that educational treatments and interventions tend to be multidimensional, variabledense, and highly context-dependent, involving consideration of much more than simple delivery of discrete behaviors.…”