How Close is Close Enough? Evaluating Propensity Score Matching Using Data from a Class Size Reduction Experiment

The ability of nonexperimental estimators to match impact estimates derived from random assignment is examined using data from the evaluation of two interdistrict magnet schools. As in previous within‐study comparisons, nonexperimental estimates differ from estimates based on random assignment when nonexperimental estimators are implemented without pretreatment measures of academic performance. With comparison groups consisting of students drawn from the same districts or districts with similar student body characteristics as the districts where treatment group students reside, using pretreatment test scores reduces the bias in nonexperimental methods between 64 and 96 percent. Adding pretreatment test scores does not achieve as much bias reduction when the comparison group consists of students drawn from districts with different student body characteristics than the treatment group students’ districts. The results suggest that using pretreatment outcome measures and comparison groups that are geographically aligned with the treatment group greatly improves the performance of nonexperimental estimators.

Section: Introductionmentioning

confidence: 66%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Can Nonexperimental Estimates Replicate Estimates Based on Random Assignment in Evaluations of School Choice? A Within‐Study Comparison

Bifulco¹

2012

“…LaLonde () and Fraker and Maynard () were the first to use this method to examine whether econometric adjustments for selection bias in an observational study could reproduce the results of job‐training RCTs. Since then, researchers have used the method to study the performance of RDD (Green et al., ; Shadish et al., ), intact group and individual case matching (Bifulco, ; Cook, Shadish, & Wong, ; Wilde & Hollister, ), and alternative strategies for covariate selection (Cook & Steiner, ). The implementation details of within‐study comparisons vary, but the basic idea is always to test the validity of a nonexperimental method by comparing its estimates to a trustworthy benchmark from an RCT.…”

Section: Introductionmentioning

confidence: 99%

Strengthening the Regression Discontinuity Design Using Additional Design Elements: A Within‐study Comparison

Wing¹,

Cook²

2013

The sharp regression discontinuity design (RDD) has three key weaknesses compared to the randomized clinical trial (RCT). It has lower statistical power, it is more dependent on statistical modeling assumptions, and its treatment effect estimates are limited to the narrow subpopulation of cases immediately around the cutoff, which is rarely of direct scientific or policy interest. This paper examines how adding an untreated comparison to the basic RDD structure can mitigate these three problems. In the example we present, pretest observations on the posttest outcome measure are used to form a comparison RDD function. To assess its performance as a supplement to the basic RDD, we designed a within‐study comparison that compares causal estimates and their standard errors for (1) the basic posttest‐only RDD, (2) a pretest‐supplemented RDD, and (3) an RCT chosen to serve as the causal benchmark. The two RDD designs are constructed from the RCT, and all analyses are replicated with three different assignment cutoffs in three American states. The results show that adding the pretest makes functional form assumptions more transparent. It also produces causal estimates that are more precise than in the posttest‐only RDD, but that are nonetheless larger than in the RCT. Neither RDD version shows much bias at the cutoff, and the pretest‐supplemented RDD produces causal effects in the region beyond the cutoff that are very similar to the RCT estimates for that same region. Thus, the pretest‐supplemented RDD improves on the standard RDD in multiple ways that bring causal estimates and their standard errors closer to those of an RCT, not just at the cutoff, but also away from it.

“…Both of these studies relied on data from the NSWD. Related studies of this type include Dehejia and Wahba (, ), Smith and Todd (), Friedlander and Robins (), Heckman and Hotz (), Heckman, Ichimura, and Todd (), Heckman et al (), Diaz and Handa (), and Wilde and Hollister ().…”

Section: Introductionmentioning

confidence: 99%

What Works Best and When: Accounting for Multiple Sources of Pureselection Bias in Program Evaluations

Jung¹,

Pirog²

2014

Most evaluations are still quasi-experimental and most recent quasi-experimental methodological research has focused on various types of propensity score matching to minimize conventional selection bias on observables. Although these methods create better-matched treatment and comparison groups on observables, the issue of selection on unobservables still looms large. Thus, in the absence of being able to run randomized controlled trials (RCTs) or natural experiments, it is important to understand how well different regression-based estimators perform in terms of minimizing pure selection bias, that is, selection on unobservables. We examine the relative magnitudes of three sources of pure selection bias: heterogeneous response bias, time-invariant individual heterogeneity (fixed effects [FEs]), and intertemporal dependence (autoregressive process of order one [AR(1)]). Because the relative magnitude of each source of pure selection bias may vary in different policy contexts, it is important to understand how well different regression-based estimators handle each source of selection bias.Expanding simulations that have their origins in the work of Heckman, LaLonde, and Smith (1999), we find that difference-in-differences (DID) using equidistant pre-and postperiods and FEs estimators are less biased and have smaller standard errors in estimating the Treatment on the Treated (TT) than other regression-based estimators. Our data analysis using the Job Training Partnership Act (JTPA) program replicates