The gold standard for whom? Schools' experiences participating in a randomised controlled trial

Abstract

Background: Randomised controlled trials (RCTs) have long been considered the gold standard in education research. Federal funds are allocated to evaluations that meet What Works Clearinghouse standards; RCT designs are required in order to meet these standards without reservations. Schools seek out interventions that are research based, in other words, interventions that have demonstrated effects in RCTs, creating a cycle that incentivises RCT research. However, RCTs pose tremendous obstacles for school-based participants.Methods: This paper presents a descriptive case study of an RCT of an adolescent reading intervention implemented in four districts over three years, drawing on interviews and document analysis to illustrate the challenges that participation in an RCT poses for schools. Results: In two participating districts, the random assignment requirement caused major difficulties with recruitment and retention. Of the 15 schools that began implementation in fall of year 1, 11 left the study by fall of year 2. Of those 11, eight explicitly stated that the random assignment requirement was the reason they withdrew. Stakeholders expressed that they felt it was unethical to withhold the intervention from some students. Conclusions: This rate of attrition casts doubt on the external validity of the programme evaluation, as these schools are likely to systematically differ from schools that met the requirements of random assignment. Suggestions are offered for alternative methods that may be used to evaluate promising interventions, in ways that are better aligned with real-world educational contexts.

Working at the intersection of research and practice: The love to read project

“…However, their input into the evaluation of the program was limited in line with plans stipulated in the original grant application. It is important to recognize that the evaluation interests and priorities of teachers and school leaders may be very different from those of researchers; failing to involve those from practice in evaluation (study design, variables of interest and actual measures) has the potential to hinder effective evaluation (Troyer, 2022) and poorly account for practice priorities. We'd therefore strongly encourage researchers to provide more scope for practice partners input into evaluation and not just the project.…”