Accounting for No-Shows in Experimental Evaluation Designs

Abstract

This article examines how to estimate the effect of a program in the presence of no- shows—persons who are assigned to the program but do not participate. The article briefly discusses the methodological problems involved, describes two current experimental evaluations that are subject to these problems, presents several estimators that overcome these problems, outlines the conditions necessary for these estimators to be feasible, and describes two extensions of the analysis that illustrate a potentially broad range of further applications.

Two years after a job loss: Long-term impact of the JOBS program on reemployment and mental health.

“…Our analyses preserved the integrity of the experimental design (Cook & Campbell, 1979) by including in the experimental condition all of the respondents who were randomized and invited to the workshop, yet 46% of them did not participate in the intervention workshop. Earlier analyses that focused on participation using instrumental variable techniques (Bloom, 1984;Little & Yau, 1998, Vinokur et ai., 1995 showed effects of participation to be about twice as large as those that are based on the conservative analyses.…”

Do Job Corps performance measures track program impacts?

“…These average impact estimates pertain to eligible applicants (which represent the combined impacts for the 73 percent of treatments who participated in the program and the 27 percent who did not). For each center performance group, we also estimated impacts for program participants only using an instrumental variables approach in which the impacts per eligible applicant were divided by the difference between the program participation rate for the treatment group and the control group crossover rate (Bloom, 1984;Angrist, Imbens, & Rubin, 1996). 5 The program participation rate did not vary across the center performance groups (it was 73 percent for treatments designated for high-and medium-performing centers, and 72 percent for those designated for low-performing centers).…”

Pragmatic trials of complex psychosocial interventions: methodological challenges

“…Other, more familiar examples will involve patients simply not turning up for their allocated therapy ('no shows', as Bloom, 1984, refers to them). Of course, nonadherence may not be all or none -some patients will turn up for a few sessions of therapy and then drop out.…”