An Empirical Comparison Between the Synthetic Control Method and HSIAO <i>et al.</i>'s Panel Data Approach to Program Evaluation

Gardeazábal, Javier; Vega-Bayo, Ainhoa

doi:10.1002/jae.2557

Cited by 64 publications

(24 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given that this is a relatively new method, there are not enough papers to formally test for specification searching (Brodeur et al, 2016;Simonsohn, Nelson, & Simmons, 2014). Also, specification searching is, of course, not something specific to the SC method, and our results do not imply that this problem is more relevant for the SC method when compared to alternative methods (Gardeazabal & Vega-Bayo, 2016). 10 In Appendix A, we present in detail sufficient conditions for this assumption.…”

Section: Synthetic Controls and Specification Searchingmentioning

confidence: 85%

Cherry Picking with Synthetic Controls

Ferman¹,

Pinto²,

Possebom³

2020

J Policy Anal Manage

179

150

View full text Add to dashboard Cite

We evaluate whether a lack of guidance on how to choose the matching variables used in the Synthetic Control (SC) estimator creates specification-searching opportunities. We provide theoretical results showing that specification-searching opportunities are asymptotically irrelevant if we restrict to a subset of SC specifications. However, based on Monte Carlo simulations and simulations with real datasets, we show significant room for specification searching when the number of pre-treatment periods is in line with common SC applications, and when alternative specifications commonly used in SC applications are also considered. This suggests that such lack of guidance generates a substantial level of discretion in the choice of the comparison units in SC applications, undermining one of the advantages of the method. We provide recommendations to limit the possibilities for specification searching in the SC method. Finally, we analyze the possibilities for specification searching and provide our recommendations in a series of empirical applications. C 2020 by the Association for Public Policy Analysis and Management on behalf of the Association for Public Policy Analysis and Management variables, the choice of how to split the pre-treatment periods into training and validation periods, and even the choice of software and data-sorting criteria (see Klößner et al., 2017, for details on this last point). Therefore, our results should be seen as a lower bound on the possibilities for specification searching in SC applications. We focus on the choice of pre-treatment outcome lags, rather than on the inclusion of covariates, because it is possible to systematically analyze the inclusion of pre-intervention outcomes lags in a way that encompasses all applications, while covariates may differ in complex ways from one application to another. We consider the possibility of specification searching in the decision to include covariates in our empirical applications. 6 Appendices are available at the end of this article as it appears in JPAM online. Go to the publisher's website and use the search engine to locate the article at http://onlinelibrary.wiley.com. 7 In Appendix B, we also consider specifications that use time-invariant covariates as predictors, in addition to functions of the pre-treatment outcomes. All results remain similar. This package solves the nested minimization problem described by equations (1) and (2). We specify the optimization method to be BFGS only and use optimization routine Low Rank Quadratic Programming when Interior Point optimization routine does not converge. 24 When we adopt this decision rule in our MC simulations, then the probability of rejecting the null at 5 percent for all specifications is lower than 1 percent in all scenarios. If we discard specifications 6 and 7, then this rejection rate ranges from 1 percent when T 0 = 12 to 2.8 percent when T 0 = 400. 25 Following the best practices in terms of transparency and replicability, Hainmueller (2014) made their dataset and replication ...

show abstract

Section: Synthetic Controls and Specification Searchingmentioning

confidence: 85%

Cherry Picking with Synthetic Controls

Ferman¹,

Pinto²,

Possebom³

2020

J Policy Anal Manage

179

150

View full text Add to dashboard Cite

show abstract

“…The data-driven SCM approach searches for optimal predictor weights in order to grant more importance to economic predictors with better predictive power. Properties of the SCM estimator, like (asymptotic) unbiasedness, have been developed by Abadie et al (2010) , while Gardeazabal and Vega-Bayo (2017 ) find that the SCM estimator performs well as compared to alternative panel approaches.…”

Section: Introductionmentioning

confidence: 99%

Comparative politics and the synthetic control method revisited: a note on Abadie et al. (2015)

Klößner

Kaul

Pfeifer

et al. 2018

Swiss J Economics Statistics

View full text Add to dashboard Cite

Recently, Abadie et al. (Am J Polit Sci 59:495–510, 2015) have expanded synthetic control methods by the so-called cross-validation technique. We find that their results are not being reproduced when alternative software packages are used or when the variables’ ordering within the dataset is changed. We show that this failure stems from the cross-validation technique relying on non-uniquely defined predictor weights. While the amount of the resulting ambiguity is negligible for the main application of Abadie et al. (Am J Polit Sci 59:495–510, 2015), we find it to be substantial for several of their robustness analyses. Applying well-defined, standard synthetic control methods reveals that the authors’ results are particularly driven by a specific control country, the USA.

show abstract

“…The quasiexperimental method, which is especially useful for estimating effects in case studies of unique interventions in small samples panel data, is called the synthetic control method. Under ideal conditions, it outperforms both the interrupted time series and the difference-in-differences methods in terms of internal validity 4. It is now widely considered one of the most credible quasiexperimental approaches for policy evaluation in political science and economics, especially when coupled with supplementary analyses in the form of sensitivity and placebo tests 5.…”

Section: Introductionmentioning

confidence: 99%