The Impacts of Reading Recovery at Scale: Results From the 4-Year i3 External Evaluation

Abstract

Reading Recovery is an example of a widely used early literacy intervention for struggling first-grade readers, with a research base demonstrating evidence of impact. With funding from the U.S. Department of Education’s i3 program, researchers conducted a 4-year evaluation of the national scale-up of Reading Recovery. The evaluation included an implementation study and a multisite randomized controlled trial with 6,888 participating students in 1,222 schools. The goal of this study was to understand whether the impacts identified in prior rigorous studies of Reading Recovery could be replicated in the context of a national scale-up. The findings of this study reaffirm prior evidence of Reading Recovery’s immediate impacts on student literacy and support the feasibility of successfully scaling up an effective intervention.

Optimal Design of Cluster- and Multisite-Randomized Studies Using Fallible Outcome Measures

“…A recent evaluation of the Reading Recovery intervention provides an exemplar two-level MSRT with a latent outcome at Level 1 (e.g., Sirinides, Gray, & May, 2018). Schools are randomly selected to ensure representativeness and then students are randomized to the treatment condition within these schools.…”

“…The primary outcome of interest is reading achievement, a latent construct estimated by the Total Reading standard score from the Iowa Tests of Basic Skills. It is likely that the reading achievement score outcome includes some degree of measurement error with reliability of these tests typically around .8–.9 (Sirinides et al, 2018).…”

“…Based on Sirinides, Gray, and May (2018), let us assume a large sample of students and schools to investigate the scalability of the Reading Recovery program and a specific subsample of 3,400 students from 1,100 rural schools (i.e., ≈ 3 per school). We predict a Reading Recovery treatment effect of δ j = .1 , student-level variance on the standardized reading test of σ Y 2 = .9 , and variance of the Reading Recovery treatment effect across schools of τ δ| 2 = .3 .…”

“…For an MSRT with a large sample size, the more important design consideration is often MDE. Using the parameter values described above but with the full sample from Sirinides et al (2018; n 1 = 6 , 800 and n 2 = 1 , 200 ), the MDE is .081 and .084 when considering unreliability in the reading test scores. Again, increases to the MDE are minimized by the high reliability of the standardized tests scores and to a lesser degree the larger individual sample size.…”

“…Last, we can consider planning this type of evaluation under an optimal sampling framework. Sirinides et al (2018) note the high cost of the intensive Reading Recovery program that entails specialized staff training and serves relatively few students at a time. As far as research costs, let us assume sampling a school costs around US$3,000, while the cost to sample a student costs around US$1,000 ( c 2 / c 1 = 3 / 1 ) with a budget of US$4 million (Sirinides et al, 2018).…”