Interactive identification of individuals with positive treatment effect while controlling false discoveries

Abstract

Out of the participants in a randomized experiment with anticipated heterogeneous treatment effects, is it possible to identify which ones have a positive treatment effect, even though each has only taken either treatment or control but not both? While subgroup analysis has received attention, claims about individual participants are more challenging. We frame the problem in terms of multiple hypothesis testing: we think of each individual as a null hypothesis (the potential outcomes are equal, for example) and aim to identify individuals for whom the null is false (the treatment potential outcome stochastically dominates the control, for example). We develop a novel algorithm that identifies such a subset, with nonasymptotic control of the false discovery rate (FDR). Our algorithm allows for interaction -a human data scientist (or a computer program acting on the human's behalf) may adaptively guide the algorithm in a data-dependent manner to gain high identification power. We also propose several extensions: (a) relaxing the null to nonpositive effects, (b) moving from unpaired to paired samples, and (c) subgroup identification. We demonstrate via numerical experiments and theoretical analysis that the proposed method has valid FDR control in finite samples and reasonably high identification power.

Evidence-Based Policy Learning

“…Moreover, unlike the FDR literature we construct algorithms for optimizing the out-of-sample power of the test. Arguably closest to our work from the FDR literature is the concurrent work of [13], which instantiates the FDR line of work on the heterogeneous treatment effect problem and presents an interactive algorithm that controls the false discovery rate of the constructed sub-group to be treated. On the contrary our work focuses on finding an assignment in a non-interactive manner, which maximizes the power of a test on a separate sample.…”