Benefits and costs of matching prior to a Difference in Differences analysis when parallel trends does not hold

Abstract

The Difference in Difference (DiD) estimator is a popular estimator built on the "parallel trends" assumption, or an assertion that the treatment group, absent treatment, would change "similarly" to the control group over time. To increase the plausibility of this assumption, a natural idea is to match treated and control units prior to a DiD analysis. In this paper, we characterize the bias of matching prior to a DiD analysis under a linear structural model. Our framework allows for both observed and unobserved confounders that have time varying effects. Given this framework, we find that matching on baseline covariates reduces the bias associated with these covariates, when compared to the original DiD estimator. We further find that additionally matching on the pre-treatment outcomes has both cost and benefit. First, it mitigates the bias associated with unobserved confounders, since matching on pre-treatment outcomes partially balances these unobserved confounders. This reduction is proportional to the reliability of the outcome, a measure of how coupled the outcomes are with these latent covariates. On the other hand, we find that matching on the pre-treatment outcome undermines the second "difference" in a DiD estimate by forcing the treated and control group's pre-treatment outcomes to be equal. This injects bias into the final estimate, analogous to the case when parallel trends holds. We extend our bias results to multivariate confounders with multiple pre-treatment periods and find similar results. Finally, we provide heuristic guidelines to practitioners on whether to match prior to their DiD analysis, along with a method for roughly estimating the reduction in bias. We illustrate our guidelines by reanalyzing a recent empirical study that used matching prior to a DiD analysis to explore the impact of principal turnover on student achievement. We find that the authors' decision to match on the pre-treatment outcomes was crucial in making the estimated treatment effect more credible.

The Long-Term Causal Effects of Winning an Erc Grant

“…First, it allows us to deal with multiple time periods and takes into account the fact that treatment e↵ects may vary with the length of exposure to the treatment. Second, it allows us to condition on covariates when the parallel trend assumption potentially holds only after conditioning on observed pre-treatment characteristics (see Ham and Miratrix 2022).…”