When comparing two distributions, it is often helpful to learn at which quantiles or values there is a statistically significant difference. This provides more information than the binary "reject" or "do not reject" decision of a global goodness-of-fit test. Framing our question as multiple testing across the continuum of quantiles τ ∈ (0, 1) or values r ∈ R, we show that the Kolmogorov-Smirnov test (interpreted as a multiple testing procedure) achieves strong control of the familywise error rate. However, its well-known flaw of low sensitivity in the tails remains. We provide an alternative method that retains such strong control of familywise error rate while also having even sensitivity, i.e., equal pointwise type I error rates at each of n → ∞ order statistics across the distribution. Our one-sample method computes instantly, using our new formula that also instantly computes goodness-of-fit p-values and uniform confidence bands. To improve power, we also propose stepdown and pre-test procedures that maintain control of the asymptotic familywise error rate. One-sample and two-sample cases are considered, as well as extensions to regression discontinuity designs and conditional distributions. Simulations, empirical examples, and code are provided.JEL classification: C12, C14, C21

Using and extending fractional order statistic theory, we characterize the O(n −1 ) coverage probability error of the previously proposed confidence intervals for population quantiles using L-statistics as endpoints in Hutson (1999). We derive an analytic expression for the n −1 term, which may be used to calibrate the nominal coverage level to get O n −3/2 [log(n)] 3 coverage error. Asymptotic power is shown to be optimal. Using kernel smoothing, we propose a related method for nonparametric inference on conditional quantiles. This new method compares favorably with asymptotic normality and bootstrap methods in theory and in simulations. Code is provided for both unconditional and conditional inference.JEL classification: C21

No abstract

We provide novel, high-order accurate methods for non-parametric inference on quantile differences between two populations in both unconditional and conditional settings. These quantile differences correspond to (conditional) quantile treatment effects under (conditional) independence of a binary treatment and potential outcomes. Our methods use the probability integral transform and a Dirichlet (rather than Gaussian) reference distribution to pick appropriate L-statistics as confidence interval endpoints, achieving highorder accuracy. Using a similar approach, we also propose confidence intervals/sets for vectors of quantiles, interquantile ranges and differences of linear combinations of quantiles. In the conditional setting, when smoothing over continuous covariates, optimal bandwidth and coverage probability rates are derived for all methods. Simulations show that the new confidence intervals have a favourable combination of robust accuracy and short length compared with existing approaches. Detailed steps for confidence interval construction are provided in online Appendix E as supporting information, and code for all methods, simulations and empirical examples is provided.

We describe our framework, deployed at Facebook, that accounts for interference between experimental units through cluster-randomized experiments. We document this system, including the design and estimation procedures, and detail insights we have gained from the many experiments that have used this system at scale. We introduce a cluster-based regression adjustment that substantially improves precision for estimating global treatment effects as well as testing for interference as part of our estimation procedure. With this regression adjustment, we find that imbalanced clusters can better account for interference than balanced clusters without sacrificing accuracy. In addition, we show how logging exposure to a treatment can be used for additional variance reduction. Interference is a widely acknowledged issue with online field experiments, yet there is less evidence from real-world experiments demonstrating interference in online settings. We fill this gap by describing two case studies that capture significant network effects and highlight the value of this experimentation framework.