Program Evaluation and Research Designs

Abstract

This chapter provides a selective review of some contemporary approaches to program evaluation. One motivation for our review is the recent emergence and increasing use of a particular kind of "program" in applied microeconomic research, the so-called Regression Discontinuity (RD) Design of Thistlethwaite and Campbell (1960). We organize our discussion of these various research designs by how they secure internal validity: in this view, the RD design can been seen as a close "cousin" of the randomized experiment. An important distinction which emerges from our discussion of "heterogeneous treatment effects" is between ex post (descriptive) and ex ante (predictive) evaluations; these two types of evaluations have distinct, but complementary goals. A second important distinction we make is between statistical statements that are descriptions of our knowledge of the program assignment process and statistical statements that are structural assumptions about individual behavior. Using these distinctions, we examine some commonly employed evaluation strategies, and assess them with a common set of criteria for "internal validity", the foremost goal of an ex post evaluation. In some cases, we also provide some concrete illustrations of how internally valid causal estimates can be supplemented with specific structural assumptions to address "external validity": the estimate from an internally valid "experimental" estimate can be viewed as a "leading term" in an extrapolation for a parameter of interest in an ex ante evaluation. * We are grateful to Diane Alexander and Pauline Leung, who provided outstanding research assistance. We thank Orley Ashenfelter, David Card, Damon Clark, Nicole Fortin, Thomas Lemieux, Enrico Moretti, Phil Oreopolous, Zhuan Pei, Chris Taber, Petra Todd, John Van Reenen, and Ken Wolpin for helpful suggestions, comments, and discussions.

Evaluating spatial policies

“…Such Randomised Control Trials (RCTs) are often considered the 'gold standard' of evaluation (Banerjee and Duflo 2010, DiNardo and Lee, 2010, Haynes et al 2012. Properly implemented, randomisation ensures that treatment and control groups are comparable, thus identifying the causal impact of policy (Katz et al 2001, Kling et al 2005, Sanbonmatsu et al 2012.…”

Identification of Models of the Labor Market

“…For more complete summaries see Heckman and Robb (1986), Heckman, Lalonde and Smith (1999), Vytlacil (2007a,2007b), Abbring and Heckman (2007), or Imbens and Wooldridge (2009). 13 DiNardo and Lee (2010) provide a discussion that is complementary to ours. Our goal in this section is not to survey the whole literature but provide a brief summary and to put it into the context of identification of the Generalized Roy…”

“…Thus we avoid a discussion of matching estimators. See Heckman, Lalonde, and Smith (1999), Imbens andWooldridge (2009), or Dinardo andLee (2010) for discussion.…”

Causal Inference in Urban and Regional Economics

“…Analysis of this simple case is a straightforward point of departure as it is well understood in the statistics literature going back to Rubin's (1974) classic treatment, and discussed extensively in Holland (1986), and in the economics literature going back to Roy (1951). Because the recovery of causal relationships in environments with binary treatment environments are also discussed at length in DiNardo & Lee's (2011) Handbook of Labor Economics chapter, we leave the development of many details to them. Indeed, much of our mission in this chapter is to extend their discussion of various empirical identification strategies to environments in which the treatment is continuous and the data are spatially indexed.…”