Nonexperimental Versus Experimental Estimates of Earnings Impacts

Abstract

To assess nonexperimental (NX) evaluation methods in the context of welfare, job training, and employment services programs, the authors reexamined the results of twelve case studies intended to replicate impact estimates from an experimental evaluation by using NX methods. They found that the NX methods sometimes came close to replicating experimentally derived results but often produced estimates that differed by policy-relevant margins, which the authors interpret as estimates of bias. Although the authors identified several study design factors associated with smaller discrepancies, no combination of factors would consistently eliminate discrepancies. Even with a large number of impact estimates, the positive and negative bias estimates did not always cancel each other out. Thus, it was difficult to identify an aggregation strategy that consistently removed bias while answering a focused question about earnings impacts of a program. They conclude that although the empirical evidence from this literature can be used in the context of training and welfare programs to improve NX research designs, it cannot on its own justify the use of such designs.

Best Practices in Quasi–Experimental Designs: Matching Methods for Causal Inference

“…However, some general guidance can be obtained. Glazerman et al (2003) find that one-to-one propensity score matching performs better than other propensity score matching methods or non-propensity score matching and that standard econometric selection correction procedures, such as instrumental variables or the Heckman selection correction, tend to perform poorly. As discussed earlier, their results also show that combining methods, such as matching and covariance adjustment, is better than using those methods individually.…”

“…The matching method reduces large covariate bias between the treated and control groups, and the regression is used to adjust for any small residual biases and to increase efficiency. These "bias-corrected" matching methods have been found by Abadie and Imbens (2006) and Glazerman, Levy, and Myers (2003) to work well in practice, using simulated and actual data. Rubin (1973bRubin ( , 1979, Rubin and Thomas (2000), and Ho et al (2007) show that models based on matched data are much less sensitive to model misspecification and more robust than are models fit in the full data sets.…”

“…Glazerman et al (2003) summarize the results from 12 case studies that attempted to replicate experimental estimates using nonexperimental data, all in the context of job training, welfare, and employment programs with earnings as the outcome of interest. The nonexperimental methods include matching and covariance adjustment.…”

Assessing ‘what works’ in international development: meta-analysis for sophisticated dummies

“…When comparing average differences across multiple randomised and non-randomised studies using meta-analysis, the evidence is mixed. One prominent article found significant differences in estimated effects between 12 replicated randomised and non-randomised studies (Glazerman et al 2003), whereas others have suggested that the differences in results are almost zero overall, whether or not the individual study differences are themselves zero Wilson 1993, Heinsman andShadish 1996). Hansen et al (2011) note, based on an assessment of within-study comparisons, that appropriate knowledge of the participation decision process is key to estimating unbiased effects.…”

Weighting Regressions by Propensity Scores

“…This includes use of propensity scores in weighted regression and in earlier techniques, such as (i) creating match sets or (ii) computing weighted contrasts between treatment and control groups. See Other references of interest include Arceneaux, Gerber, and Green (2006), Glazerman, Levy, and Myers (2003), Peikes, Moreno, and Orzol (2008), Wilde and Hollister (2007). These authors point to serious weaknesses in the propensity-score methods that have been used for program evaluation.…”