False discovery rate control for effect modification in observational studies

Interactive identification of individuals with positive treatment effect while controlling false discoveries

“…Although our interactive procedure is designed for FDR control at an individual level, we propose extensions to FDR control at a subgroup level in Appendix A.1. As a brief summary, Karmakar et al (2018) propose to control FDR subgroup by constructing a p-value for each subgroup and apply the classical BH method (Benjamini and Hochberg, 1995). While their method has many orthogonal benefits (e.g., handling observational studies), it is not trivially applicable to control FDR at an individual level, because their p-values would only take value 1/2 or 1 when each subgroup has exactly one subject, leading to zero identification power following the BH procedure.…”

“…Karmakar et al (2018); Gu and Shen (2018); Xie et al (2018) discuss FDR control at a subgroup level, where the latter two have little discussion on incorporating continuous covariates and require parametric assumptions on the outcomes. Thus, we follow the setup in Karmakar et al (2018) to compare the FDR control at a subgroup level (in their paper) and individual level (in our paper). Let the subgroups be non-overlapping sets {G 1 , .…”

“…In this section, we show that the idea of interactive testing can be extended to control FDR on subgroup level, where we aim at identifying multiple subgroups with positive effects and upper bounding the proportion of falsely identified subgroups. Recall that FDR control at a subgroup level is studied by Karmakar et al (2018) as we review in Section 1.2 of the main paper.…”

“…An interactive algorithm to identify subgroups. We first follow the same steps of Karmakar et al (2018) to define subgroups and generate the p-value for each subgroup. Specifically, the subgroups G g for g ∈ [G] is defined using the outcomes and covariates {Y j , X j } n j=1 (by an arbitrary algorithm or strategy, such as grouping subjects with the same covariates).…”

“…For each subgroup G g , we can compute a p-value P g by the classical Wilcoxon test (or using a permutation test, which obtains the null distribution by permuting the treatment assignment {A i } n i=1 ). The interactive procedure we propose differs from Karmakar et al (2018) by how we process the p-values of the subgroups. We adopt the work of Lei et al (2020) that proposes an interactive procedure with FDR control for generic multiple testing problems.…”