What To Do (and Not to Do) with Causal Panel Analysis under Parallel Trends: Lessons from A Large Reanalysis Study

Chiu, Albert; X, Lan; Liu, Ziyi; Xu, Yiqing

doi:10.2139/ssrn.4490035

Cited by 7 publications

(5 citation statements)

References 84 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our study provides a number of important insights, some of which resonate with the conclusions of other recent reviews (e.g., Chiu et al 2023;Freedman et al 2023;Roth et al 2023). First, we highlight that the recently identified problems of TWFE are primarily not due to the estimator itself.…”

Section: Discussionsupporting

confidence: 83%

“…There is now a bewildering list of alternative estimators that tackle the above-mentioned issue (for formal reviews see Chiu et al 2023;Roth et al 2023). In this study we will discuss and compare five alternatives to TWFE.…”

Section: Dynamic Did Estimatorsmentioning

confidence: 99%

“…In Figure 8, we introduce non-parallel trends while keeping no anticipation and no omitted variable biases. The parallel trends assumption is the most consequential assumption for all DiD-like methods, which may often be violated in practice (Chiu et al 2023).…”

Section: Non-parallel Trendsmentioning

confidence: 99%

“…Against this background of the growing literature of new DiD estimators, a recent reanalysis of 37 published studies that relied on DiD showed that the simple TWFE and the novel DiD estimators often yielded effectively the same conclusions (Chiu et al 2023). The question then arises as to how serious the newly discovered shortcomings of TWFE are in applied research.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

When Can We Use Two-Way Fixed-Effects (TWFE): A Comparison of TWFE and Novel Dynamic Difference-in-Differences Estimators

Rüttenauer,

Aksoy

2024

Preprint

View full text Add to dashboard Cite

The conventional Two-Way Fixed-Effects (TWFE) estimator has been under strain lately. Recent literature has revealed potential shortcomings of TWFE when the treatment effects are heterogeneous. Scholars have developed new advanced dynamic Difference-in-Differences (DiD) estimators to tackle these potential shortcomings. However, confusion remains in applied research as to when the conventional TWFE is biased and what issues the novel estimators can and cannot address. In this study, we first provide an intuitive explanation of the problems of TWFE and elucidate the key features of the novel alternative DiD estimators. We then systematically demonstrate the conditions under which the conventional TWFE is inconsistent. We employ Monte Carlo simulations to assess the performance of dynamic DiD estimators under violations of key assumptions, which likely happens in applied cases. While the new dynamic DiD estimators offer notable advantages in capturing heterogeneous treatment effects, we show that the conventional TWFE performs generally well if the model specifies an event-time function. All estimators are equally sensitive to violations of the parallel trends assumption, anticipation effects or violations of time-varying exogeneity. Despite their advantages, the new dynamic DiD estimators tackle a very specific problem and they do not serve as a universal remedy for violations of the most critical assumptions. We finally derive, based on our simulations, recommendations for how and when to use TWFE and the new DiD estimators in applied research.

show abstract

Section: Discussionsupporting

confidence: 83%

Section: Dynamic Did Estimatorsmentioning

confidence: 99%

Section: Non-parallel Trendsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

When Can We Use Two-Way Fixed-Effects (TWFE): A Comparison of TWFE and Novel Dynamic Difference-in-Differences Estimators

Rüttenauer,

Aksoy

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…After much effort and hundreds of hours of work, we are able to replicate the main results of 67 articles. 6 The low replication rate is consistent with what is reported in Hainmueller, Mummolo, and Xu (2019) and Chiu et al (2023). The main reasons for failures of replication are incomplete data (38 articles), incomplete code or poor documentation (4 articles), and replication errors (5 articles).…”

Section: Datasupporting

confidence: 71%

How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice Based on 67 Replicated Studies

Lal,

Lockhart,

et al. 2024

Polit. Anal.

Self Cite

View full text Add to dashboard Cite

Instrumental variable (IV) strategies are widely used in political science to establish causal relationships, but the identifying assumptions required by an IV design are demanding, and assessing their validity remains challenging. In this paper, we replicate 67 articles published in three top political science journals from 2010 to 2022 and identify several concerning patterns. First, researchers often overestimate the strength of their instruments due to non-i.i.d. error structures such as clustering. Second, IV estimates are often highly uncertain, and the commonly used t-test for two-stage-least-squares (2SLS) estimates frequently underestimate the uncertainties. Third, in most replicated studies, 2SLS estimates are significantly larger in magnitude than ordinary-least-squares estimates, and their absolute ratio is inversely related to the strength of the instrument in observational studies—a pattern not observed in experimental ones—suggesting potential violations of unconfoundedness or the exclusion restriction in the former. We provide a checklist and software to help researchers avoid these pitfalls and improve their practice.

show abstract

What Can We Learn about the Effects of Democracy Using Cross-National Data?

DOUCETTE

2024

Am Polit Sci Rev

View full text Add to dashboard Cite

More than 1,100 studies have been published that examine the effects of democracy using cross-national data since 2000. This article examines whether these analyses have sufficient statistical power to detect an effect of democracy. Using Monte Carlo simulation and examining consensus effects previously reported in the literature, the article finds that studies are only powered to detect very strong effects of democracy when examining countries over time. This raises questions about what sort of relationships can be analyzed using cross-national data.

show abstract

What To Do (and Not to Do) with Causal Panel Analysis under Parallel Trends: Lessons from A Large Reanalysis Study

Cited by 7 publications

References 84 publications

When Can We Use Two-Way Fixed-Effects (TWFE): A Comparison of TWFE and Novel Dynamic Difference-in-Differences Estimators

When Can We Use Two-Way Fixed-Effects (TWFE): A Comparison of TWFE and Novel Dynamic Difference-in-Differences Estimators

How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice Based on 67 Replicated Studies

What Can We Learn about the Effects of Democracy Using Cross-National Data?

Contact Info

Product

Resources

About