Rethinking the Tradeoffs of Within-Subjects Designs in International Relations Experiments

Abstract

All experimentalists in International Relations face a choice over their designs—including that between between-subjects and within-subjects designs. This manuscript helps them make informed decisions in two ways. First, we reassess existing findings on the advantages of within-subjects designs, such as higher statistical power and precision in treatment effect estimates. Second, we contextualize the tradeoffs of within-subjects designs by examining their external validity across countries. Our results are based on a unique survey experiment with within-subjects and between-subjects designs in Brazil, China, Japan, and Sweden and a meta-analysis of 26 paired within-subjects conditions from existing lab experiments. We find that within-subjects designs provide higher statistical power, precision in treatment effect estimates, insights on treatment effect heterogeneity, and strong external validity across countries. However, we find evidence of order effects in within-subjects experiments. Our results imply that within-subjects designs are a practical option in International Relations—particularly for under-resourced researchers—albeit with tradeoffs.