Making Every Study Count: Learning From Replication Failure to Improve Intervention Research

Abstract

Why, when so many educational interventions demonstrate positive impact in tightly controlled efficacy trials, are null results common in follow-up effectiveness trials? Using case studies from literacy, this article suggests that replication failure can surface hidden moderators—contextual differences between an efficacy and an effectiveness trial—and generate new hypotheses and questions to guide future research. First, replication failure can reveal systemic barriers to program implementation. Second, it can highlight for whom and in what contexts a program theory of change works best. Third, it suggests that a fidelity first and adaptation second model of program implementation can enhance the effectiveness of evidence-based interventions and improve student outcomes. Ultimately, researchers can make every study count by learning from both replication success and failure to improve the rigor, relevance, and reproducibility of intervention research.

Fluency Interventions for Elementary Students with Reading Difficulties: A Synthesis of Research from 2000–2019

“…Generally, fluency interventions examined, which constituted mostly repeated reading procedures, made positive contributions to gains in reading fluency and reading comprehension among children with reading difficulties. Considering that failure to replicate findings presents as a salient issue in research [92,93], the replication of findings of previous syntheses despite not including single-case studies and only focusing on those studies that included a control group provides convergent evidence for the importance of fluency on reading outcomes. Second, extending from previous syntheses, our finding compared relative gains of fluency interventions on fluency and reading comprehension outcomes.…”

Using Implementation Fidelity to Aid in Interpreting Program Impacts: A Brief Review

“…Eight studies had null results but high fidelity, suggesting that the program design and contextual factors outlined by Jacob et al (this issue, pp. 580–589) and Kim (this issue, pp. 599–607) may play a role in producing program outcomes.…”

Expecting and Learning From Null Results

“…For example, Kim (this issue, pp. 599–607) points to research on Collaborative Strategic Reading (CSR).…”

“…As a result, it is a near certainty that studies will vary from one another along multiple dimensions, and we really have no way of knowing which dimension(s) might be responsible for the differences in the observed effects. For example, Kim (this issue, pp. 599–607) pointed out that there were implementation differences in Vaughn et al (2011) and Hitchcock et al (2011), and certainly this seems like a plausible reason why results might differ (see Hill & Erickson, this issue, pp.…”