Synthetic learner: model-free inference on treatments over time

Abstract

Understanding of the effect of a particular treatment or a policy pertains to many areas of interest -ranging from political economics, marketing to health-care and personalized treatment studies. In this paper, we develop a non-parametric, model-free test for detecting the effects of treatment over time that extends widely used Synthetic Control tests. The test is built on counterfactual predictions arising from many learning algorithms. In the Neyman-Rubin potential outcome framework with possible carry-over effects, we show that the proposed test is asymptotically consistent for stationary, beta mixing processes. We do not assume that class of learners captures the correct model necessarily. We also discuss estimates of the average treatment effect, and we provide regret bounds on the predictive performance. To the best of our knowledge, this is the first set of results that allow for example any Random Forest to be useful for provably valid statistical inference in the Synthetic Control setting. In experiments, we show that our Synthetic Learner is substantially more powerful than classical methods based on Synthetic Control or Difference-in-Differences, especially in the presence of non-linear outcome models.

Synthetic Control As Online Linear Regression

“…In parallel, there is a rapidly growing methodological literature providing statistical guarantees for synthetic control methods. 1 Existing guarantees for synthetic control are typically derived under a linear factor model or a vector autoregressive model of the outcomes (among others, Abadie et al, 2010;Ben-Michael et al, 2019Ferman and Pinto, 2021;Viviano and Bradic, 2019). 2 While the guarantees formally hold under these outcome models, there is a wide sense of optimism that the synthetic control method is robust to these modeling assumptions.…”

“…These regret guarantees average performance over time, which correspond to expected loss under a design assumption of random treatment timing. This paper is perhaps closest to Viviano and Bradic (2019). They propose an ensemble scheme to aggregate predictions from multiple predictive models, which can include synthetic control, interactive fixed effect models, and random forests.…”

“…They propose an ensemble scheme to aggregate predictions from multiple predictive models, which can include synthetic control, interactive fixed effect models, and random forests. Using results from the online learning literature, Viviano and Bradic (2019)'s ensemble scheme has the no-regret property, making the ensemble predictions competitive against the prediction of any fixed predictive model in the ensemble. Under sampling processes that yield good performance for some predictive model in the ensemble, Viviano and Bradic (2019) then derive performance guarantees for the ensemble learner.…”

“…Using results from the online learning literature, Viviano and Bradic (2019)'s ensemble scheme has the no-regret property, making the ensemble predictions competitive against the prediction of any fixed predictive model in the ensemble. Under sampling processes that yield good performance for some predictive model in the ensemble, Viviano and Bradic (2019) then derive performance guarantees for the ensemble learner. In contrast, we study synthetic control directly in the worst-case setting, and connect guarantees in the worst-case setting to guarantees on statistical risk in a design-based framework.…”

Estimating the effectiveness of permanent price reductions for competing products using multivariate Bayesian structural time series models

“…Bread sales are unlikely to be impacted by the price of cookies; instead, they capture daily and weekly trends, which are useful in modeling how the sales of cookies would have evolved had we not reduced their price. More broadly, these methods have been successfully applied to evaluate the effectiveness of policy changes in healthcare (Kreif et al, 2016;Papadogeorgou et al, 2018;Viviano and Bradic, 2019), economics (Billmeier and Nannicini, 2013;Abadie et al, 2015;Dube and Zipperer, 2015;Gobillon and Magnac, 2016;Ben-Michael et al, 2018), marketing and online advertising (Brodersen et al, 2015;Li, 2019), amongst others.…”