The Internal and External Validity of the Regression Discontinuity Design: A Meta‐analysis of 15 Within‐study Comparisons

Abstract

Theory predicts that regression discontinuity (RD) provides valid causal inference at the cutoff score that determines treatment assignment. One purpose of this paper is to test RD's internal validity across 15 studies. Each of them assesses the correspondence between causal estimates from an RD study and a randomized control trial (RCT) when the estimates are made at the same cutoff point where they should not differ asymptotically. However, statistical error, imperfect design implementation, and a plethora of different possible analysis options, mean that they might nonetheless differ. We test whether they do, assuming that the bias potential is greater with RDs than RCTs. A second purpose of this paper is to investigate the external validity of RD by exploring how the size of the bias estimates varies across the 15 studies, for they differ in their settings, interventions, analyses, and implementation details. Both Bayesian and frequentist meta‐analysis methods show that the RD bias is below 0.01 standard deviations on average, indicating RD's high internal validity. When the study‐specific estimates are shrunken to capitalize on the information the other studies provide, all the RD causal estimates fall within 0.07 standard deviations of their RCT counterparts, now indicating high external validity. With unshrunken estimates, the mean RD bias is still essentially zero, but the distribution of RD bias estimates is less tight, especially with smaller samples and when parametric RD analyses are used.

Evidence‐based juvenile justice programs and practices: A critical review

“…This requires that the covariates used in a study capture the true process of selection into treatment that is correlated with the study outcome. However, in actual research practice with QEDs, only regression discontinuity designs have been shown to demonstrably meet this condition (Chaplin et al., 2018; Cook, Shadish, & Wong, 2008). It is unlikely that the QEDs in the set of studies included in Lipsey's meta‐analysis met this assumption.…”

Effectiveness of pre-operative anaemia screening and increased Tranexamic acid dose on outcomes following unilateral primary, elective total hip or knee replacement: a statistical analysis plan for an interrupted time series and regression discontinuity study

“…The local nature of the effect examined in RDD can also be used in optimising threshold levels. In this case we may be able to examine if a threshold Hb of 120 or 130g/L may be more appropriate for females, as is being suggested in some studies 11 , 26 – 28 . As the TXA policy affects all patients it is only possible to conduct an RDD analysis for the anaemia screening programme, using data since the inception of this programme (1 st February 2013, Figure 1 ).…”

Within study comparisons and risk of bias in international development: Systematic review and critical appraisal

“…MDRC Steiner and Assessing correspondence between experimental and nonexperimental results in within-study-comparisons EdPolicyWorks Working Paper Wong and Steiner (2016) Designs of empirical evaluations of nonexperimental methods in field settings EdPolicyWorks Working Paper Jaciw (2016) Assessing the accuracy of generalized inferences from comparison group studies using a within-study comparison approach: The methodology Evaluation Review Wong et al (2017) Empirical performance of covariates in education observational studies Methodological Studies Chaplin et al (2018) The internal and external validity of the regression discontinuity design: A meta-analysis of 15 within-study-comparisons Policy Analysis and Management methodological designs, such as RDD (Chaplin et al, 2018; and propensity score matching (PSM; Shadish, 2013), or map internal replication designs . Only one known review is dedicated to evidence from social and economic development programmes in L&MICs (Hansen et al, 2013).…”

“…This included: Review of existing narrative reviews of internal replication studies (e.g., Cook et al, ; Hansen et al, ; Wong, Valentine, & Miller‐Bains, ) and meta‐analyses of these studies (e.g., Chaplin et al, ; Glazerman et al, ). Systematic electronic and hand‐searches for internal replication studies in international development. Critical appraisal (risk of bias assessment) in benchmark trials. Calculation of standardised bias estimates and narrative analysis of differences in effect sizes between the benchmark and nonrandomised QE study arms. Research objective 2: development of a risk of bias tool in nonrandomised studies of interventions (ROBINS‐I) for assessing risk of bias in RDDs.…”