Using Implementation Fidelity to Aid in Interpreting Program Impacts: A Brief Review

Abstract

Poor program implementation constitutes one explanation for null results in trials of educational interventions. For this reason, researchers often collect data about implementation fidelity when conducting such trials. In this article, we document whether and how researchers report and measure program fidelity in recent cluster-randomized trials. We then create two measures—one describing the level of fidelity reported by authors and another describing whether the study reports null results—and examine the correspondence between the two. We also explore whether fidelity is influenced by study size, type of fidelity measured and reported, and features of the intervention. We find that as expected, fidelity level relates to student outcomes; we also find that the presence of new curriculum materials positively predicts fidelity level.

Putting PjBL to the Test: The Impact of Project-Based Learning on Second Graders’ Social Studies and Literacy Learning and Motivation in Low-SES School Settings

“…In our research, the degree of consistency between teachers' practice and the curriculum materials turned out to be consequential for students' learning and motivation, as has been found in many past studies (see Hill & Erickson, 2019, for a review). Overall, our intervention did not have a statistically significant impact on students' informational writing.…”

“…Observation data indicated that teachers randomly assigned to the comparison group did not implement project-based-learning (mean score of 1.1 on the 1–3 scale described earlier). Collectively, these data reflect several types of fidelity identified by Hill and Erickson (2019). Put another way, the experiment tested what it was designed to test.…”

“…That is, the mean difference between experimental and comparison groups in social studies was 0.482 standard deviations even after controlling for baseline scores. In line with relevant methodological research (e.g., Ho et al, 2007), the Institute of Education Sciences What Works PjBL Impact Clearinghouse (2014) considers an ES of 0.25 or higher to be ''substantively important'' (p. 23). See Table 3.…”

Expecting and Learning From Null Results

“…For example, Kim (this issue, pp. 599–607) pointed out that there were implementation differences in Vaughn et al (2011) and Hitchcock et al (2011), and certainly this seems like a plausible reason why results might differ (see Hill & Erickson, this issue, pp. 590–598).…”

Commentary on the Null Results Special Issue

“…In the second article, Heather Hill and Anna Erickson present a meta-analysis of fidelity, predictors of fidelity, and associations between fidelity and impacts (Hill & Erickson, this issue, pp. 590–598).…”