A Call for Cautious Interpretation of Meta-Analytic Reviews

Abstract

Meta-analytic reviews collect available empirical studies on a specified domain and calculate the average effect of a factor. Educators as well as researchers exploring a new domain of inquiry may rely on the conclusions from meta-analytic reviews rather than reading multiple primary studies. This article calls for caution in this regard because the outcome of a meta-analysis is determined by how effect sizes are calculated, how factors are defined, and how studies are selected for inclusion. Three recently published meta-analyses are reexamined to illustrate these issues. The first illustrates the risk of conflating effect sizes from studies with different design features; the second illustrates problems with delineating the variable of interest, with implications for cause-effect relations; and the third illustrates the challenge of determining the eligibility of candidate studies. Replication attempts yield outcomes that differ from the three original meta-analyses, suggesting also that conclusions drawn from meta-analyses need to be interpreted cautiously.

The Relationship Between Utterance and Perceived Fluency: A Meta‐Analysis of Correlational Studies

“…The literature search identified 5,061 articles, published from 1943 to 2019. Following methodological guidelines for meta‐analysis (Boers et al., 2020; Moher et al., 2009; Plonsky & Oswald, 2015), their titles, abstracts, and study descriptors (e.g., keywords, subject categories) were then inspected to see if (a) the study measured any aspects of oral fluency in any form, and (b) speech data were produced by L2 learners. A sample of approximately 2% ( k = 100) of the 5,061 studies was independently examined by the first and third authors.…”

“…Several methodological limitations of our meta‐analysis need to be acknowledged to avoid overinterpretation of our findings. First, the total number of primary studies was relatively small, because of our strict screening procedure, which is crucial for the robustness of findings from meta‐analyses (Boers et al., 2020). Meanwhile, the number of effect sizes in some subgroups in moderator analyses was too small to perform some subgroup analyses.…”

Relationships Between Phonological Awareness and Reading in Spanish: A Meta‐Analysis

“…All these precautions and the exhaustive selection of the studies to be included were necessary for the sake of replicability of the meta‐analysis as indicated by Boers, Bryfonski, Faez, and McKay (2020).…”

“…Last, as mentioned by Boers et al. (2020), the outcome of a meta‐analysis is determined by how effect sizes are calculated. Therefore, the choice of including zero‐order correlations (which reflect only the relationship between two variables) and partial correlations (which also control for other variables) between PA and reading might have influenced the overall mean effect sizes.…”

Glossing and vocabulary learning

“…This is puzzling because it contradicts the inclusion criteria stipulated by the authors. I will therefore refrain from saying more about Vahedi et al's meta-analysis in this article, apart from cautioning readers that meta-analytic reviews need to be interpreted cautiously, just like any other research report (e.g., Boers et al, 2020).…”