To assess nonexperimental (NX) evaluation methods in the context of welfare, job training, and employment services programs, the authors reexamined the results of twelve case studies intended to replicate impact estimates from an experimental evaluation by using NX methods. They found that the NX methods sometimes came close to replicating experimentally derived results but often produced estimates that differed by policy-relevant margins, which the authors interpret as estimates of bias. Although the authors identified several study design factors associated with smaller discrepancies, no combination of factors would consistently eliminate discrepancies. Even with a large number of impact estimates, the positive and negative bias estimates did not always cancel each other out. Thus, it was difficult to identify an aggregation strategy that consistently removed bias while answering a focused question about earnings impacts of a program. They conclude that although the empirical evidence from this literature can be used in the context of training and welfare programs to improve NX research designs, it cannot on its own justify the use of such designs.

We estimate school-choice preferences revealed by the rank-ordered lists submitted by more than 22,000 applicants to a citywide lottery for more than 200 traditional and charter public schools in Washington, D.C. The results confirm previously reported findings that commuting distance, school demographics, and academic indicators play important roles in school choice and that there is considerable heterogeneity of preferences. Higher and lower income choosers respond to academic quality measures, but respond to different indicators of quality. Simulations suggest segregation by race and income would be reduced and enrollment in high-performing schools increased if policymakers were to relax school capacity constraints in individual campuses. The simulations also suggest that removing the lowest performing schools as choice options could further reduce segregation and increase enrollment in high-performing schools.

Abstract

This paper reports on a randomized experiment to study the impact of an alternative teacher preparation program, Teach for America (TFA), on student achievement and other outcomes. We found that TFA teachers had a positive impact on math achievement and no impact on reading achievement. The size of the impact on math scores was about 15 percent of a standard deviation, equivalent to about one month of instruction. The general conclusions did not differ substantially for subgroups of teachers, including novice teachers, or for subgroups of students. We found no impacts on other student outcomes such as attendance, promotion, or disciplinary incidents, but TFA teachers were more likely to report problems with student behavior than were their peers. The findings contradict claims that such programs allowing teachers to bypass the traditional route to the classroom harm students. © 2006 by the Association for Public Policy Analysis and Management

No abstract