An Equivalence Approach to Balance and Placebo Tests

Abstract

Recent emphasis on credible causal designs has led to the expectation that scholars justify their research designs by testing the plausibility of their causal identification assumptions, often through balance and placebo tests. Yet current practice is to use statistical tests with an inappropriate null hypothesis of no difference, which can result in equating nonsignificant differences with significant homogeneity. Instead, we argue that researchers should begin with the initial hypothesis that the data are inconsistent with a valid research design, and provide sufficient statistical evidence in favor of a valid design. When tests are correctly specified so that difference is the null and equivalence is the alternative, the problems afflicting traditional tests are alleviated. We argue that equivalence tests are better able to incorporate substantive considerations about what constitutes good balance on covariates and placebo outcomes than traditional tests. We demonstrate these advantages with applications to natural experiments.

Mobilize for Our Lives? School Shootings and Democratic Accountability in U.S. Elections

“…16 Using equivalence testing, we can confidently rule out effects as small as 3 percentage points in all specifications (and in most specifications we can rule out effects smaller than that). Figure A14 shows that our models are able to rule out Hartman and Hidalgo's (2018) meaningful effect sizes in all cases. In short, school shootings have (at best) small/nonconsequential positive effects on voter turnout, but more than likely school shootings have no effect at all.…”

“…To put this into perspective, only 26 total counties (0.2%) in the sample period had an election within this margin. Figure A13 shows that our models are able to rule out Hartman and Hidalgo's (2018) meaningful effect sizes in all cases. For almost all elections, school shootings' effects are totally inconsequential in determining the election, if they have any effect at all.…”

“…These vary some, but not much, across our subgroups, with all groups having somewhere between 91% and 96% (Cohen's standard) and 95-100%-Hartman and Hidalgo's (2018) small or nonsubstantive effect sizes. Given that most of the theories revolving around the effect of shootings anticipate a mobilizing/positive effect, it is also important to note that only 2/147 effects (1.4%) clear the (adjusted) significance levels, are positive, and are substantively meaningful (by Hartman and Hidalgo's [2018] standard). 15 Finally, larger tragedies with more fatalities are just as likely to elicit negative effects as positive effects.…”

Family Ties? The Limits of Fathering Daughters on Congressional Behavior

“…However, it is difficult to support the claim that having a daughter has "no effect" based on a nonsignificant regression coefficient alone. Therefore, we follow the suggestions of others to rule out that the range of potential estimates contains large, substantively meaningful effects (Gross, 2014;Hartman & Hidalgo, 2018;Lakens, 2017;Rainey, 2014). We calculate 90% confidence intervals for the coefficient estimate of interest in our model (Rainey, 2014).…”

Daughters do not affect political beliefs in a new democracy

“…effect" against the alternative hypothesis of "no effect," referred to as equivalence testing. We employ a two-one sided test (TOST) approach (Robinson and Froese, 2004;Lakens, 2017;Hartman and Hidalgo, 2018), and calculate a formal p-value by specifying a theoretical effect size λ, any smaller than which we would consider a small or approximately zero treatment effect. We then test the null hypothesis that the effect is greater than or equal to λ, or less than or equal to −λ.…”