Toward a Science of Failure Analysis: A Narrative Review

Abstract

When a researcher tests an educational program, product, or policy in a randomized controlled trial and detects a significant effect on an outcome, the intervention is usually classified as something that “works.” When expected effects are not found, there is seldom an orderly and transparent analysis of plausible reasons why. Accumulating and learning from possible failure mechanisms is not standard practice in education research, and it is not common to design interventions with causes of failure in mind. This chapter develops Boruch and Ruby’s proposition that the education sciences would benefit from a systematic approach to the study of failure. We review and taxonomize recent reports of large-scale randomized controlled trials in K–12 schooling that yielded at least one null or negative major outcome, including the nature of the event and reasons (if provided) for why it occurred. Our purpose is to introduce a broad framework for thinking about educational interventions that do not produce expected effects and seed a cumulative knowledge base on when, how, and why interventions do not reach expectations. The reasons why an individual intervention fails to elicit an outcome are not straightforward, but themes emerge when researchers’ reports are synthesized.

Using pupils’ grade obtained in national examinations as an outcome measure in evaluations: some considerations for the design of randomised controlled trials

“…A potential contributory factor to the underpowering of trials comes from the choice of primary outcome. Previous research has pointed to the selection of outcomes that are too distal to the intervention or treatment, leading to an effective loss of power (Allen-Platt et al, 2021;Jacob et al, 2019). In this paper, we focus on the nature of a particular outcome indicator frequently used in education trials -summary examination grade -that can act as a contributory factor to the underpowering of studies, with particular reference to the situation in England.…”