Oracle and adaptive false discovery rate controlling methods for one‐sided testing: theory and application in treatment effect evaluation

Abstract

Summary Economists are often interested in identifying effective policies or treatments together with subpopulations of individuals who respond positively (or with a sign that is expected) to these treatment interventions. In this paper, we propose an optimal false discovery rate controlling method that is especially useful for such one‐sided testing problems. The proposed procedure is optimal in the sense of minimizing the false non‐discovery rate while controlling the false discovery rate at a pre‐specified level; it uses a deconvolution method based on non‐parametric maximum likelihood estimation, which allows for a broader class of treatment effect distributions than existing methods do. The proposed test demonstrates good small‐sample performance in Monte Carlo simulations and it is applied to study the effect of attending a more selective high school in Romania. The application reveals strong evidence of treatment effect heterogeneity, in that students who marginally gain access to higher‐ranked schools are more likely to benefit if the higher‐ranked school has a relatively high admission score cut‐off – or, in other words, is more selective.

Viewpoint: The human capital approach to inference

“…Another approach is to develop optimal statistical decision rules in the presence of heterogeneous treatment effects (Sun and McLain ). Gu and Shen () show that this approach can be applied to estimating the heterogeneous effect of attending a more selective school, though, it is limited to one‐sided testing. A promising direction is to extend optimal statistical decision rules to cases where the sign of the treatment effect is not known.…”

Viewpoint: The Human Capital Approach to Inference

“…Another approach is to develop optimal statistical decision rules in the presence of heterogeneous treatment effects (Sun and McLain 2012). Gu and Shen (2016) show that this approach can be applied to estimating the heterogeneous effect of attending a more selective school, though, it is limited to one-sided testing. A promising direction is to extend optimal statistical decision rules to cases where the sign of the treatment effect is not known.…”

Multiple Testing and the Distributional Effects of Accountability Incentives in Education

“…Note that in our application the number of hypotheses being tested is quite small particularly relative to genomic studies from genome wide association studies. If the number of hypotheses were large it is well known that FWER controlling procedures typically have low power, and in responseGu and Shen (2017) propose an optimal false discovery rate controlling method.…”