Strengthening the Regression Discontinuity Design Using Additional Design Elements: A Within‐study Comparison

Abstract

The sharp regression discontinuity design (RDD) has three key weaknesses compared to the randomized clinical trial (RCT). It has lower statistical power, it is more dependent on statistical modeling assumptions, and its treatment effect estimates are limited to the narrow subpopulation of cases immediately around the cutoff, which is rarely of direct scientific or policy interest. This paper examines how adding an untreated comparison to the basic RDD structure can mitigate these three problems. In the example we present, pretest observations on the posttest outcome measure are used to form a comparison RDD function. To assess its performance as a supplement to the basic RDD, we designed a within‐study comparison that compares causal estimates and their standard errors for (1) the basic posttest‐only RDD, (2) a pretest‐supplemented RDD, and (3) an RCT chosen to serve as the causal benchmark. The two RDD designs are constructed from the RCT, and all analyses are replicated with three different assignment cutoffs in three American states. The results show that adding the pretest makes functional form assumptions more transparent. It also produces causal estimates that are more precise than in the posttest‐only RDD, but that are nonetheless larger than in the RCT. Neither RDD version shows much bias at the cutoff, and the pretest‐supplemented RDD produces causal effects in the region beyond the cutoff that are very similar to the RCT estimates for that same region. Thus, the pretest‐supplemented RDD improves on the standard RDD in multiple ways that bring causal estimates and their standard errors closer to those of an RCT, not just at the cutoff, but also away from it.

Alternatives to Randomized Control Trials: A Review of Three Quasi-experimental Designs for Causal Inference

“…Next, the authors discussed the Regression Discontinuity design which assigns participants to treatment and control conditions based on a just and defensible cut point on an assignment variable and subsequently measure the discontinuity of the treatment and control trends. The inference of this design becomes much stronger when utilizing the pre-post framework outlined by Wing and Cook (2013), making RD comparable to an RCT. Lastly, the authors discussed Propensity Score Matching, which pairs control and treatment participants on the similarity of their scores to account for selection bias.…”

“…To remedy these limitations, Wing and Cook (2013) propose the addition of a pretest comparison group. The reasoning for using pretest scores is to provide information about the relationship between the cut point and outcome prior to treatment.…”

“…An unknown smoothing function is represented by the g (A i ), and it is assumed to be constant across the pre-and posttest (for further discussion of smoothing parameters see Peng, 1999). Wing and Cook (2013) used the data from the Cash and Counseling Demonstration RCT (Dale & Brown, 2007) to test the efficacy of a pre-post RD design. In the original study, disabled Medicaid beneficiaries were randomly assigned to obtain two types of healthcare services to examine the differences on a variety of health, social, and economic outcomes.…”

Quasi-Experimental Designs

“…They compared RCT estimates of the former with RDD estimates of the latter, finding general agreement between the two. Wing and Cook (2013) propose adding a pretest measure of the outcome or a comparison group to the RDD. Using a WSC, they find that this can improve the precision, internal, and external validity of RDD estimates.…”

Evaluating Econometric Evaluations of Post-Secondary Aid

“…Findings to date are detailed in our working paper Angrist et al (2014). 1 Our focus here is methodological: in the spirit of LaLonde's (1986) pioneering comparison of job training effects from randomized and non-experimental analyses, and the recent Wing and Cook (2013) within-study evaluation of a RD design, we compare our experimental results with covariate-controlled estimates from a pre-experimental cohort and with RD estimates from the experimental sample. The results show covariates do little to mitigate selection bias, but RD estimates that exploit institutional idiosyncrasies in the award process come close to an appropriately-defined experimental benchmark.…”