On the reliability of published findings using the regression discontinuity design in political science

Abstract

The regression discontinuity (RD) design offers identification of causal effects under weak assumptions, earning it the position as a standard method in modern political science research. But identification does not necessarily imply that the causal effects can be estimated accurately with limited data. In this paper, we highlight that estimation is particularly challenging with the RD design and investigate how these challenges manifest themselves in the empirical literature. We collect all RD-based findings published in top political science journals from 2009-2018. The findings exhibit pathological features; estimates tend to bunch just above the conventional level of statistical significance. A reanalysis of all studies with available data suggests that researcher's discretion is not a major driver of these pathological features, but researchers tend to use inappropriate methods for inference, rendering standard errors artificially small. A retrospective power analysis reveals that most of these studies were underpowered to detect all but large effects. The issues we uncover, combined with well-documented selection pressures in academic publishing, cause concern that many published findings using the RD design are exaggerated, if not entirely spurious.

War-Time Military Service Can Affect Partisan Preferences

“…Alternatively, the negative effect we identify might resemble a more durable form of preference change that is stable even years after the war was concluded. Since our data is constructed of repeated cross-sectional surveys rather than a panel survey, and since any given cross-sectional survey alone is underpowered to identify our main estimates (Stommes et al, 2021), we are not ideally set up to fully address the question of effect durability.…”

Contested Killings: The Mobilizing Effects of Community Contact with Police Violence

“…Our formal RDiT models are estimated using the rdrobust (Calonico, Cattaneo, and Titiunik 2015) package in R, which allows for the estimation of nonparametric bandwidths (i.e., a bandwidth around the cutpoint that is driven by the inherent properties of the data and not selected by the researcher), bias-corrected point estimates, and robust standard errors. We thus sidestep the major problems recently associated with RDD designs (Stommes, Aronow, and Sävje 2021). We use a local polynomial of 1 to avoid over-fitting the data.…”

The How and Why of Bayesian Nonparametric Causal Inference

“…In particular, Gaussian process methods had interval coverage almost always at or near the desired level. This is important, as recent work has found that most existing methods that are used for RDD designs tend to understate uncertainty and provide anti-conservative inference (Stommes et al, 2021). This work has been further extended to spatial RDD settings where the goal is estimation of the treatment effect curve on a two-dimensional boundary (Rischard et al, 2020).…”