A Theory of Experimenters

Abstract

This paper proposes a decision-theoretic framework for experiment design. We model experimenters as ambiguity-averse decision-makers, who make trade-offs between subjective expected performance and robustness. This framework accounts for experimenters' preference for randomization, and clarifies the circumstances in which randomization is optimal: when the available sample size is large enough or robustness is an important concern. We illustrate the practical value of such a framework by studying the issue of rerandomization. Rerandomization creates a trade-off between subjective performance and robustness. However, robustness loss grows very slowly with the number of times one randomizes. This argues for rerandomizing in most environments.

Understanding and misunderstanding randomized controlled trials

“…The tension between randomization and precision that goes back to Fisher, Gosset, and Savage has been reopened in recent papers by Kasy (2016), Banerjee et al (BCS) (2016) and Banerjee et al (BCMS) (2017).…”

“…Savage also notes that several people with different priors may be involved in an investigation and that individual priors may be unreliable because of “vagueness and temptation to self-deception,” defects that randomization may alleviate, or at least evade. BCMS (2017) provide a proof of a Bayesian no-randomization theorem, and BCS (2016) provide an illustration of a school administrator who has long believed that school outcomes are determined, not by school quality, but by parental background, and who can learn the most by placing deprived children in (supposed) high-quality schools and privileged children in (supposed) low-quality schools, which is the kind of study setting to which case study methodology is well attuned. As BCS note, this allocation would not persuade those with different priors, and they propose randomization as a means of satisfying skeptical observers.…”

The Perry Preschoolers at Late Midlife: A Study in Design-Specific Inference

“…Our approach is more broadly applicable to other compromised social science experiments, including those using re-randomization designs. See Bruhn and McKenzie (2009), Morgan and Rubin (2012), Li et al (2018), and Banerjee et al (2019Banerjee et al ( , 2017. rate of our procedure in practice with that of the standard inferential methods, such as asymptotic, bootstrap, and permutation tests, used in previous studies that ignore essential details of the experimental setup.…”

Editor in Chief’s Comment: External Validity in Systematic Reviews

“…5 To fix ideas, suppose that sites like this one means serving the subgroup that this site would serve. Then, a meta-analytic perspective (e.g., Hedges & Olkin, 2014) would suggest modeling the impact of some program model, p, on subgroup, g, as follows:…”