When second best is good enough: a comparison between a true experiment and a regression discontinuity quasi-experiment

Berk, Richard A.; Barnes, Geoffrey C.; Ahlman, Lindsay; Kurtz, Ellen S.

doi:10.1007/s11292-010-9095-3

Cited by 76 publications

(65 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We might offer that in such types of limited datasets that scholars consider alternative strategies for causal identification which do not rely on the selection on observables assumption, such as instrumental variables (Angrist 2006) or regression discontinuity designs (Berk et al 2010). Our results also would seem to support the belief that propensity score methods will work best on datasets which include a rich array of pre-intervention measures which can be used to establish equivalence over a wide range of items across multiple domains.…”

Section: Discussionmentioning

confidence: 57%

Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores

et al. 2015

View full text Add to dashboard Cite

Objectives Propensity score methods rely on an untestable assumption of unconfoundedness for making causal inference. Yet, empirical applications using propensity scores in criminology routinely invoke this assumption without careful scrutiny. Methods We use a dataset with a wide range of observable, potential confounders, which allows us to evaluate recidivism outcomes for adolescent offenders who are sentenced to either placement or probation. We then systematically withhold important known confounders from the matching process to demonstrate the effectiveness of sensitivity checks in sizing up the robustness of these treatment effect estimates in the case where hidden biases clearly exist. Results We find important variability in the estimated treatment effect, and a large degree of imbalance in 'unobserved' covariates, which we did not explicitly control for. The hidden biases observed in our controlled analysis would have at least been suggested in an actual application by the low gamma statistics that attended our analysis, a statistic that is not reported in most criminological applications of propensity score analysis. Conclusions Researchers who use propensity score methods should openly discuss potential limitations of their analysis due to hidden bias and report bias sensitivity checks based on the gamma statistic when statistically significant treatment effects are reported.

show abstract

Section: Discussionmentioning

confidence: 57%

Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores

et al. 2015

View full text Add to dashboard Cite

show abstract

“…Six studies provided us with standardized RD bias estimates directly or with the pooled standard deviation of the original study outcome necessary to calculate it directly (Aiken et al., ; Gleason, Resch, & Berk, ; Kisbu‐Sakarya, Cook, & Tang, in press; Nickerson, ; Shadish et al., ; Tang, Cook, & Kisbu‐Sakarya, forthcoming). Four studies used binary outcomes (Barrera‐Osorio, Filmer, & McIntyre, ; Berk et al., ; Green et al., ; Hyytinen et al., ). While it is common to use log odds ratios for binary outcomes in meta‐analyses, for ease of interpretation we stuck with mean differences of the binary outcomes and calculated treatment and control group standard deviations for them as the square roots of their variances.…”

Section: Methods For Meta‐analyzing Wsc Estimatesmentioning

confidence: 99%

The Internal and External Validity of the Regression Discontinuity Design: A Meta‐analysis of 15 Within‐study Comparisons

Chaplin¹,

Cook²,

Zurovac³

et al. 2018

J Policy Anal Manage

View full text Add to dashboard Cite

Theory predicts that regression discontinuity (RD) provides valid causal inference at the cutoff score that determines treatment assignment. One purpose of this paper is to test RD's internal validity across 15 studies. Each of them assesses the correspondence between causal estimates from an RD study and a randomized control trial (RCT) when the estimates are made at the same cutoff point where they should not differ asymptotically. However, statistical error, imperfect design implementation, and a plethora of different possible analysis options, mean that they might nonetheless differ. We test whether they do, assuming that the bias potential is greater with RDs than RCTs. A second purpose of this paper is to investigate the external validity of RD by exploring how the size of the bias estimates varies across the 15 studies, for they differ in their settings, interventions, analyses, and implementation details. Both Bayesian and frequentist meta‐analysis methods show that the RD bias is below 0.01 standard deviations on average, indicating RD's high internal validity. When the study‐specific estimates are shrunken to capitalize on the information the other studies provide, all the RD causal estimates fall within 0.07 standard deviations of their RCT counterparts, now indicating high external validity. With unshrunken estimates, the mean RD bias is still essentially zero, but the distribution of RD bias estimates is less tight, especially with smaller samples and when parametric RD analyses are used.

show abstract

“…condition, whenever her/his observed value of the assignment variable equals or exceeds (resp. As proven in Goldberger (2008), the RDD can empirically produce causal effect estimates that are similar to those estimates of a standard randomized study (Aiken et al 1998;Buddelmeyer and Skoufias 2004;Black et al 2005;Schochet 2009;Berk et al 2010;Shadish et al 2011). Under relatively mild conditions, notably when subjects have imperfect control of the assignment variable, the RDD provides a "locally-randomized experiment."…”

Section: Discussionmentioning

confidence: 79%

Nonparametric Bayesian Inference in Biostatistics

Mitra

Müeller²

2015

View full text Add to dashboard Cite

We briefly review some of the nonparametric Bayesians models that are most widely used in biostatistics and bioinformatics. We define the Dirichlet process, Dirichlet process mixtures, the Polya tree, the dependent Dirichlet process and the Gaussian process prior. These few models and variations cover a major part of the models that are used in the literature. The discussion includes references to variations of the basic models that are defined in the chapters of this volume.

show abstract

When second best is good enough: a comparison between a true experiment and a regression discontinuity quasi-experiment

Cited by 76 publications

References 37 publications

Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores

Evolutionary regression? Assessing the problem of hidden biases in criminal justice applications using propensity scores

The Internal and External Validity of the Regression Discontinuity Design: A Meta‐analysis of 15 Within‐study Comparisons

Nonparametric Bayesian Inference in Biostatistics

Contact Info

Product

Resources

About