The Adequacy of Comparison Group Designs for Evaluations of Employment-Related Programs

Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators?

“…His paper played an important role in the late 1980's movement towards using experiments to evaluate social programs (see, e.g., Burtless and Orr, 1986, and Burtless, 1995). Fraker and Maynard (1987) perform a similar analysis that focuses more on comparison group selection than LaLonde (1986) and reach similar conclusions. Heckman and Hotz (1989) respond to the LaLonde (1986) study by applying a broader range of specification tests to guide the choice among nonexperimental estimators.…”

“…Under choice-based sampling, weights are required to consistently estimate the probabilities of program participation. 21 When the weights are unknown, Heckman and Todd (1995) show that with a slight modification, matching methods can still be applied, because the odds ratio estimated using the incorrect weights (i.e., ignoring the fact that samples are choice-based) is a scalar multiple of the true odds ratio, which is itself a monotonic transformation of the propensity scores. Therefore, matching can proceed on the (misweighted) estimate of the odds ratio (or of the log odds ratio).…”

Sensitivity of matching-based program evaluations to the availability of control variables

“…typical programs using typical assignment rules). So far, most of the existing literature uses social experiments conducted in the U.S. 3 to obtain a benchmark estimate of the effect of interest (LaLonde, 1986, Fraker and Maynard, 1987, Friedlander and Robins, 1995, Dehejia and Wahba, 1999Heckman, Ichimura, and Todd, 1997, Heckman, Ichimura, Smith, and Todd, 1998, and Smith and Todd 2005. 4 These studies match the experimental data to another nonexperimental dataset and then compare the result using the experimental control group with the results using the non-experimental control group.…”

Back to Work: Testing Reemployment Services for Displaced Workers

“…Although not published until much later, analyses of the major problems with nonexperimental methods used to evaluate CETA were being circulated at the time (Fraker and Maynard 1987;LaLonde 1986;LaLonde and Maynard 1987).…”