A Theory of Experimenters: Robustness, Randomization, and Balance

Abstract

This paper studies the problem of experiment design by an ambiguity-averse decision-maker who trades off subjective expected performance against robust performance guarantees. This framework accounts for real-world experimenters’ preference for randomization. It also clarifies the circumstances in which randomization is optimal: when the available sample size is large and robustness is an important concern. We apply our model to shed light on the practice of rerandomization, used to improve balance across treatment and control groups. We show that rerandomization creates a trade-off between subjective performance and robust performance guarantees. However, robust performance guarantees diminish very slowly with the number of rerandomizations. This suggests that moderate levels of rerandomization usefully expand the set of acceptable compromises between subjective performance and robustness. Targeting a fixed quantile of balance is safer than targeting an absolute balance objective. (JEL C90, D81)

Banerjee, Duflo, Kremer, and the Rise of Modern Development Economics*

“…Both of these design innovations have become the pattern for scores of new studies on a huge range of topics, and these innovations continue. To name just a few recent examples, Duflo has been working with econometricians on how to use innovations in machine learning to predict heterogeneous treatment effects from RCTs (Chernozhukov et al ., ), and Banerjee has been working with other theorists to articulate a theory that explains why randomization is so convincing, and exploring implications for experimental design (Banerjee et al ., ).…”

“…Both of these design innovations have become the pattern for scores of new studies on a huge range of topics, and these innovations continue. To name just a few recent examples, Duflo has been working with econometricians on how to use innovations in machine learning to predict heterogeneous treatment effects from RCTs (Chernozhukov et al, 2018), and Banerjee has been working with other theorists to articulate a theory that explains why randomization is so convincing, and exploring implications for experimental design (Banerjee et al, 2020). While I have framed the experimental revolution in modern development economics in the discussion above as, in some sense, a reaction to the challenges of approaching the development economics problem from a macroeconomic perspective, it is also crucial to note that it builds substantially on other work that was happening at the same time in the microeconomics of development.…”

Strategic Sample Selection

“…Tetenov (2016) analyzed an evaluator's optimal commitment to a decision rule when privately informed researchers select into costly testing. Banerjee et al (2020) analyzed experiment design by an ambiguity‐averse researcher facing an adversarial evaluator.…”

Addressing Validity and Generalizability Concerns in Field Experiments

“…A very recent literature sheds light on the normative appeal of randomized controlled trials and the underlying treatment assignment procedures. Banerjee et al (2020) show that an ambiguity averse decision maker who wants to provide robust evidence on potential treatment effects prefers pure randomization or certain forms of re-randomization compared to deterministic treatment assignment. We connect to this literature by empirically comparing several re-randomization techniques and pure randomization.…”