Three Conditions under Which Experiments and Observational Studies Produce Comparable Causal Estimates: New Findings from Within‐Study Comparisons

Abstract

This paper analyzes 12 recent within-study comparisons contrasting causal estimates from a randomized experiment with those from an observational study sharing the same treatment group. The aim is to test whether different causal estimates result when a counterfactual group is formed, either with or without random assignment, and when statistical adjustments for selection are made in the group from which random assignment is absent. We identify three studies comparing experiments and regression-discontinuity (RD) studies. They produce quite comparable causal estimates at points around the RD cutoff. We identify three other studies where the quasi-experiment involves careful intact group matching on the pretest. Despite the logical possibility of hidden bias in this instance, all three cases also reproduce their experimental estimates, especially if the match is geographically local. We then identify two studies where the treatment and nonrandomized comparison groups manifestly differ at pretest but where the selection process into treatment is completely or very plausibly known. Here too, experimental results are recreated. Two of the remaining studies result in correspondent experimental and nonexperimental results under some circumstances but not others, while two others produce different experimental and nonexperimental estimates, though in each case the observational study was poorly designed and analyzed. Such evidence is more promising than what was achieved in past within-study comparisons, most involving job training. Reasons for this difference are discussed. © 2008 by the Association for Public Policy Analysis and Management. INTRODUCTIONComprehensive program evaluation depends on validly determining a program's causal impacts. Debate has been vigorous about the role experiments and observational studies should play in identifying such impacts. The main reason for preferring experiments is that, when perfectly implemented, they create intervention and control groups that do not initially differ in expectation and so do not differ on any measured or unmeasured variables. However, the regression-discontinuity design (RD) and instrumental variables (IV) also provide unbiased causal inference in theory. So additional technical justification for preferring experiments is required. It comes from experimental estimates being more precise than RD and IV estimates (Goldberger, 1972) and also from the experiment's assumptions being more transparent in research practice. IV's main assumption is that the instrument is only correlated with outcome through treatment. This assumption is well warranted when the treatment is allocated via random assignment (Angrist, Imbens, & Rubin, 1996) or RD (Hahn, Todd, & van der Klauuw, 2001), but otherwise it is usually unclear whether the assumption is met in practice. RD's main assumption is that the functional form relating the assignment variable to outcome is fully known. This can usually be checked quite well in the data, but perhaps not quite as well as with the ...

What makes articles highly cited?

“…It is a problem that has mostly been ignored by researchers in the past and mostly plagues non-experimental research (but also experimental research too, for example, when testing mediation in the case of an endogenous mediator). If used correctly, however, causal claims can be made in non-experimental settings, whether using structural equations, two-stage, regression discontinuity, propensity score, or difference-in-differences models (see Antonakis, et al, 2010;Cook, Shadish, & Wong, 2008;Shadish & Cook, 1999;Shadish, Cook, & Campbell, 2002, for in-depth discussion).…”

“…To conclude, there are a variety of ways in which to use observational data to make clear causal inferences and interested readers should consult more technical literature on this matter (Cook, et al, 2008;Foster, 2010;James, Mulaik, & Brett, 1982;Morgan & Winship, 2007;Pearl, 2000;Rubin, 1974Rubin, , 2008Shadish & Cook, 2009;Shadish, et al, 2002;Shipley, 2000).…”

Research design influence on study outcomes in crime and justice: a partial replication with public area surveillance

“…Two studies are noteworthy. Cook et al (2008) compared estimates from randomized experiments with estimates from observational studies covering a wide range of areas (e.g., education, job training, school dropout prevention). An important feature of this research is that it involved within-study comparisons where the randomized experiments and observational studies shared treatment groups.…”

Assessing ‘what works’ in international development: meta-analysis for sophisticated dummies

“…This is one reason why their inclusion in policy-relevant meta-analyses has been highly controversial. 2 The empirical literature has shown that, for individual studies, weak methodological designs can lead to severe risks of bias in causal attribution, whereas well-conducted studies that carefully model participation (including IV, RDD and matching) can yield the same results as RCTs at an individual study level (Cook et al 2008, Hansen et al 2011. When comparing average differences across multiple randomised and non-randomised studies using meta-analysis, the evidence is mixed.…”