Confounding and regression adjustment indifference‐in‐differencesstudies

Abstract

Objective To define confounding bias in difference‐in‐difference studies and compare regression‐ and matching‐based estimators designed to correct bias due to observed confounders. Data sources We simulated data from linear models that incorporated different confounding relationships: time‐invariant covariates with a time‐varying effect on the outcome, time‐varying covariates with a constant effect on the outcome, and time‐varying covariates with a time‐varying effect on the outcome. We considered a simple setting that is common in the applied literature: treatment is introduced at a single time point and there is no unobserved treatment effect heterogeneity. Study design We compared the bias and root mean squared error of treatment effect estimates from six model specifications, including simple linear regression models and matching techniques. Data collection Simulation code is provided for replication. Principal findings Confounders in difference‐in‐differences are covariates that change differently over time in the treated and comparison group or have a time‐varying effect on the outcome. When such a confounding variable is measured, appropriately adjusting for this confounder (ie, including the confounder in a regression model that is consistent with the causal model) can provide unbiased estimates with optimal SE. However, when a time‐varying confounder is affected by treatment, recovering an unbiased causal effect using difference‐in‐differences is difficult. Conclusions Confounding in difference‐in‐differences is more complicated than in cross‐sectional settings, from which techniques and intuition to address observed confounding cannot be imported wholesale. Instead, analysts should begin by postulating a causal model that relates covariates, both time‐varying and those with time‐varying effects on the outcome, to treatment. This causal model will then guide the specification of an appropriate analytical model (eg, using regression or matching) that can produce unbiased treatment effect estimates. We emphasize the importance of thoughtful incorporation of covariates to address confounding bias in difference‐in‐difference studies.

Utilizing causal diagrams across quasi‐experimental approaches

“…A strength of a BACI design is that it already accounts for certain confounding variables. Confounding in BACI designs occurs only if a variable (1) effects the treatment group and (2) has an effect on the outcome trends , which can occur when a variable has a time‐varying difference between treatment groups or a time‐varying effect on the outcome (Zeldow & Hatfield, 2021). In our simulation, neither depth nor structural complexity have a time‐varying difference between treatment groups or a time‐varying effect on the outcome, so the application of a BACI analysis will return an accurate causal estimate for MPA of 1.07 [0.96, 1.20], without the need to adjust for these variables (Appendix S1: Section S3).…”

“…However, we can return an accurate estimate of 1.06 [0.97, 1.16] by making the appropriate adjustment for structural complexity (Appendix S1: Section S3). We refer readers to Zeldow and Hatfield (2021), who provide instructions on how to adjust for confounding variables, when they do arise in BACI studies.…”

Benefits and costs of matching prior to a Difference in Differences analysis when parallel trends does not hold

“…Given the popularity of matching prior to DiD in empirical studies, many works analyze matching prior to a DiD analysis and show that matching may actually hurt or help depending on different scenarios (Daw and Hatfield, 2018;Ding and Li, 2019;Lindner and McConnell, 2019;Zeldow and Hatfield, 2021;Chabé-Ferret, 2015. These existing works can be summarized into two categories.…”

“…The first are those that characterize the bias when either parallel trends or conditional parallel trends perfectly holds (Ding and Li, 2019;Daw and Hatfield, 2018). The second are those that characterize the bias in more general settings but only through simulations as opposed to mathematical derivations (Lindner and McConnell, 2019;Zeldow and Hatfield, 2021;Chabé-Ferret, 2015). Our work differs from existing literature in that we provide exact (as opposed to bounds) mathematical characterizations of the bias under a general framework that allows both imperfect conditional and unconditional parallel trends through unobserved and observed confounders.…”

Identifying and estimating effects of sustained interventions under parallel trends assumptions

“…In contrast, the literature on difference-in-differences (DID) largely bases identification on parallel trends assumptions rather than sequential exchangeability (Ashenfelter and Card, 1985;Lechner, 2010;Roth et al, 2022). Parallel trends posit that time trends in average potential outcomes are independent of the observed treatment (Ashenfelter and Card, 1985;Lechner, 2010;Marcus and Sant'Anna, 2021), and does not require we adjust for all confounders in the usual sense (Zeldow and Hatfield, 2021). The typical target parameter is an average treatment effect in the treated (ATT) for a treatment occurring at a single time point.…”