Expecting and Learning From Null Results

Abstract

This commentary addresses three issues raised in the articles in this issue. First, conversations about replication efforts should begin with a reasonable and agreed-upon definition of what it means to say that a study did or did not replicate the results of another study. Second, if a replication failure has been identified, using the surface similarity of the studies to reverse-engineer an explanation is unlikely to be helpful. Finally, researchers and consumers should expect small and heterogeneous effects, and this fact points to the need to think meta-analytically.

Toward a Science of Failure Analysis: A Narrative Review

“…realistic assessments of the program theory being tested, the likely counterfactual condition, and the likely size of impacts” (C. J. Hill, 2019, p. 609) and “get used to the idea that the effects of educational interventions on valued outcomes are usually going to be small” (Valentine, 2019, p. 612). Contributors counsel researchers to give more practical weight to precisely estimated null or negative effects versus imprecise estimates.…”

“…J. Hill, 2019; H. C. Hill & Erickson, 2019; Jacob et al, 2019; Kim, 2019; Valentine, 2019). And we newly add to the literature the relative frequency of reasons for failed outcomes offered by researchers themselves and strategies to minimize failure potential in the design, implementation, and analysis stages of experiments.…”

“…For instance, Arens et al (2012) point out that their intervention was not powered to detect effect sizes smaller than 0.35 standard deviations and they used only one outcome measure. Considering effect sizes as small as 0.20 or 0.30 are generally considered practically meaningful (Spybrook & Raudenbush, 2009), and some researchers allege that “small and probably heterogeneous effects” are “the facts” of education research (Valentine, 2019, p. 612), the power of Arens et al’s study is a limitation. It could be avoided by using more than one outcome measure, increasing sample size, or preventing attrition, as reductions in sample size also reduce power (Spybrook & Raudenbush, 2009).…”

“…J. Hill, 2019; H. C. Hill & Erickson, 2019; Jacob et al, 2019; Kim, 2019; Valentine, 2019). Based on our findings, we provide additional and augmented implications for researchers below.…”

“…For example, in large-scale educational RCTs, Jacob et al (2019, p. 586) point out that “rarely is there a no services option” because business as usual is some variant of the intervention itself, such as a math intervention compared to regular math instruction. Given that control students’ services so often overlap with treatment students’ services, and that student and teacher mobility and absence are known and somewhat predictable aspects of schools, a credible effect size may be less than the 0.20 MDES studies are typically powered to detect (Jacob et al, 2019; Spybrook & Raudenbush, 2009; Valentine, 2019).…”