Randomization Tests and Multi-Level Data in U.S. State Politics

Abstract

Many hypotheses in U.S. state politics research are multi-level, positing that state-level variables affect individual-level behavior. Unadjusted standard errors for state-level variables are too small, leading to overconfidence and possible false rejection of null hypotheses. Primo, Jacobsmeier, and Milyo (2007) explore this problem in their reanalysis of Wolfinger, Highton, and Mullin's (2005) data on the effects of post-registration laws on voter turnout. Primo et al. advocate the use of clustered standard errors to solve the overconfidence problem, but we offer an alternative solution: randomization tests. Randomization tests are non-parametric tests that do not rely on comparisons to theoretical test statistic distributions. Instead, they use distributions tailored to the data, created by randomly scrambling the data many times to simulate what would be observed under the null hypothesis. Unlike with clustering, with the randomization test, U.S. state-level reforms generally fail to be significant both as additive effects and as interactions with individual characteristics.

Legal Constraints on Supreme Court Decision Making: Do Jurisprudential Regimes Exist?

“…Some of the feasible tests for KR1 (and KR2) were not included in those papers (indicated by ''na'' in the table), but are performed here. 13 The third column shows the observed chisquare test statistic (with the ''after'' treatment assigned as in KR), along with the degrees of freedom (these capture how many interaction terms are being 10 This is the approach of recent social science applications (Donohue and Wolfers 2006, Erikson, Pinto, and Rader 2010, Helland and Tabarrok 2004. 11 We take 1,500 shuffles, being sure only to use those with the correct number of degrees of freedom for comparing the observed test statistic.…”

The Impact of State Medical Malpractice Reform on Individual‐Level Health Care Expenditures

“…We also conducted a series of sensitivity checks. The first was a randomization test, an approach that mitigates the bias in the estimated standard errors in DID analyses and that has been increasingly used for analyzing how state policies affect individual‐level behavior (Bertrand, Duflo, and Mullainathan ; Erikson, Pinto, and Rader ). For our randomization test, we took the observed data as given and generated a new data file by randomly assigning states to adopting a malpractice reform in a randomly selected year.…”

Goodbye Smokers’ Corner

“…More recently, Barrios et al (2012) provide a detailed theoretical and empirical explanation of why researchers should care about the data structure beyond state-level clustering. The application of RI is not restricted to DiD approaches (Erikson et al, 2010); it can also be applied within the regression discontinuity design (Cattaneo et al, 2015) or the potential outcome framework (Ho and Imai, 2006). The basic idea of such randomization tests is intuitive and very applicable due to its weak assumptions.…”

“…In our application, individuals' treatment statuses do not depend on just one variable (e.g., the state of residence), which is often the case for policy interventions (e.g., Erikson et al, 2010), but on several factors: state, year, age cohort, school track, and interview month. Thus, we perform an augmented randomization procedure by permuting all treatment-determining variables randomly and assigning the treatment status thereafter, according to such pseudo characteristics.…”

“…This simple and intuitive inferential technique gives valid p-values even for error terms which are clustered, have unknown structures or are otherwise complex. To date, randomization inference has not been applied in the smoking literature, even though it is a common method in other fields of treatment effect evaluation (Barrios et al, 2012;Cattaneo et al, 2015;Erikson et al, 2010;Ho and Imai, 2006;Small et al, 2008). Ultimately, we introduce an augmented RI permutation procedure, where we consider all the factors determining our treatment indicator.…”