How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment

Wilde, Elizabeth Ty; Hollister, Robinson G.

doi:10.1002/pam.20262

Cited by 82 publications

(56 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since only a subset of 15 Recent work published in JPAM using a simpler form of matching (Wilde & Hollister, 2007) notes that a lack of pre-program data may help explain discrepancies they find between matching estimates and experimental estimates. 16 The formula for the panel version of the propensity score matching estimator is:…”

Section: Estimation and Discussionmentioning

confidence: 99%

The effects of an employer subsidy on employment outcomes: A study of the work opportunity and welfare‐to‐work tax credits

Hamersma

2008

J Policy Anal Manage

View full text Add to dashboard Cite

Employer subsidies such as the Work Opportunity Tax Credit (WOTC) and the Welfare-to-Work Tax Credit (WtW) are designed to encourage employment by partially reimbursing employers for wages paid to certain welfare recipients and other disadvantaged workers. In this paper, I examine the effects of these subsidies on employment, wages, and job tenure using unique administrative data from Wisconsin. My ability to precisely identify the subsidy-certified workers allows me to distinguish the effects of program participation from mere eligibility. Using propensity score matching estimation, I find some evidence of short-term improvements in labor market outcomes, but little evidence of sustained benefits. © 2008 by the Association for Public Policy Analysis and Management. INTRODUCTIONEmployer subsidies are used throughout the developed world with the intention of reducing unemployment and improving labor market outcomes among those having difficulty finding jobs. The "New Deal" in the U.K., the "Plusjobb" program in Sweden, and the "Youth Employment Contract" in France are just a few examples of programs developed in recent years that provide substantial subsidies to employers for hiring certain categories of workers. Germany has been in heated debate over the introduction of similar programs that were proposed in May 2006. In the U.S., employer subsidies have been in place since the 1970s. However, there is little evidence that programs subsidizing employers are effective in improving employment outcomes for targeted workers.In this paper, I estimate the effects of the two current U.S. employer subsidy programs of this kind, called the Work Opportunity Tax Credit (WOTC) and the Welfare-to-Work Tax Credit (WtW), on employment outcomes of disadvantaged workers. These credits offer subsidies to firms that hire certain welfare recipients, disadvantaged youth, disabled individuals, and others who may have difficulty in the labor market. Past work on similar employer-based credits found weak or even nonexistent employment effects, which convinced Congress to allow these earlier subsidies to expire in 1994. Despite this decision, new programs were implemented in 1996 (WOTC) and 1998 (WtW) and have been reauthorized several times, at a cost of up to $500 million per year, even though there has not yet been a careful study of their effectiveness.Most past evaluations of employer tax credits similar to the WOTC and WtW have been limited by an inability to identify program participants. These studies typically Sarah HamersmaThe Effects of an Employer Subsidy on Employment Outcomes: A Study of the Work Opportunity and Welfare-to-Work Tax Credits The Effects of an Employer Subsidy on Employment Outcomes / 499Journal of Policy Analysis and Management DOI: 10.1002/pam Published on behalf of the Association for Public Policy Analysis and Management used survey data (either firm-or worker-level) that included information on eligibility without identifying which workers were actually certified (approved) for the subsidies, le...

show abstract

Section: Estimation and Discussionmentioning

confidence: 99%

The effects of an employer subsidy on employment outcomes: A study of the work opportunity and welfare‐to‐work tax credits

Hamersma

2008

J Policy Anal Manage

View full text Add to dashboard Cite

show abstract

“…Of course, this last step is a fail-safe, last resort technique, as fixing data problems by collecting better data is generally preferred to fixing them with assumption-based statistics after the fact (Wilde & Hollister, 2007). And valid randomization is still the only technique known to be able to avoid confounding from variables not measured or related to those matched on or adjusted for.…”

Section: Parametric Adjustmentmentioning

confidence: 99%

A “politically robust” experimental design for public policy evaluation, with application to the Mexican Universal Health Insurance program

King

Gakidou

Ravishankar

et al. 2007

J Policy Anal Manage

114

118

View full text Add to dashboard Cite

We develop an approach to conducting large-scale randomized public policy experiments intended to be more robust to the political interventions that have ruined some or all parts of many similar previous efforts. Our proposed design is insulated from selection bias in some circumstances even if we lose observations; our inferences can still be unbiased even if politics disrupts any two of the three steps in our analytical procedures; and other empirical checks are available to validate the overall design. We illustrate with a design and empirical validation of an evaluation of the Mexican Seguro Popular de Salud (Universal Health Insurance) program we are conducting. Seguro Popular, which is intended to grow to provide medical care, drugs, preventative services, and financial health protection to the 50 million Mexicans without health insurance, is one of the largest health reforms of any country in the last two decades. The evaluation is also large scale, constituting one of the largest policy experiments to date and what may be the largest randomized health policy experiment ever. © 2007 by the Association for Public Policy Analysis and Management

show abstract

“…10 The paper by Wilde and Hollister (2007) is one of the papers reviewed by Cook et al (2008), and they claim that because Wilde and Hollister control on too few covariates and draw their comparison group from other areas than where the treatment group resides, the Wilde and Hollister paper does not offer a good test of propensity score matching.…”

Section: Can Nonexperimental Methods Replicate Experimental Findings?mentioning

confidence: 99%

Setting up social experiments: the good, the bad, and the ugly

Barnow

2010

ZAF

View full text Add to dashboard Cite

Standard-Nutzungsbedingungen:Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden.Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen.Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Abstract It is widely agreed that randomized controlled trials -social experiments -are the gold standard for evaluating social programs. There are, however, many important issues that cannot be tested using social experiments, and often things go wrong when conducting social experiments. This paper explores these issues and offers suggestions on ways to deal with commonly encountered problems. Social experiments are preferred because random assignment assures that any differences between the treatment and control groups are due to the intervention and not some other factor; also, the results of social experiments are more easily explained and accepted by policy officials. Experimental evaluations often lack external validity and cannot control for entry effects, scale and general equilibrium effects, and aspects of the intervention that were not randomly assigned. Experiments can also lead to biased impact estimates if the control group changes its behavior or if changing the number selected changes the impact. Other problems with conducting social experiments include increased time and cost, and legal and ethical issues related to excluding people from the treatment. Things that sometimes go wrong in social experiments include programs cheating on random assignment, and participants and/or staff not understanding the intervention rules. The random assignment evaluation of the Job Training Partnership Act in the United States is used as a case study to illustrate the issues. Terms of use: Documents in

show abstract

How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment

Cited by 82 publications

References 18 publications

The effects of an employer subsidy on employment outcomes: A study of the work opportunity and welfare‐to‐work tax credits

The effects of an employer subsidy on employment outcomes: A study of the work opportunity and welfare‐to‐work tax credits

A “politically robust” experimental design for public policy evaluation, with application to the Mexican Universal Health Insurance program

Setting up social experiments: the good, the bad, and the ugly

Contact Info

Product

Resources

About