Comparing distributions by multiple testing across quantiles or CDF values

Abstract

When comparing two distributions, it is often helpful to learn at which quantiles or values there is a statistically significant difference. This provides more information than the binary "reject" or "do not reject" decision of a global goodness-of-fit test. Framing our question as multiple testing across the continuum of quantiles τ ∈ (0, 1) or values r ∈ R, we show that the Kolmogorov-Smirnov test (interpreted as a multiple testing procedure) achieves strong control of the familywise error rate. However, its well-known flaw of low sensitivity in the tails remains. We provide an alternative method that retains such strong control of familywise error rate while also having even sensitivity, i.e., equal pointwise type I error rates at each of n → ∞ order statistics across the distribution. Our one-sample method computes instantly, using our new formula that also instantly computes goodness-of-fit p-values and uniform confidence bands. To improve power, we also propose stepdown and pre-test procedures that maintain control of the asymptotic familywise error rate. One-sample and two-sample cases are considered, as well as extensions to regression discontinuity designs and conditional distributions. Simulations, empirical examples, and code are provided.JEL classification: C12, C14, C21

Non‐parametric inference on (conditional) quantile differences and interquantile ranges, using L‐statistics

“…Future research could use our framework to derive CIs with shorter length rather than the equal‐tailed/median‐unbiased property. Related work on inference on distributions and quantile marginal effects is found in Goldman and Kaplan () and Kaplan (), respectively. A formal extension to regression discontinuity as in the set‐up of Calonico et al.…”

“…Because the CIs have joint CP, this procedure has strong control of the familywise error rate at level α as defined in, e.g., Lehmann and Romano (, Section 9.1): the probability of falsely rejecting at least one true is below α, regardless of which are true. Goldman and Kaplan () extend this approach to distributional inference where , including step‐down and pre‐test procedures to improve power.…”

“…The related idea of even pointwise type I error rates is applied to inference on the entire distribution by Goldman and Kaplan ().…”

distcomp: Comparing distributions

“…This section contains additional theoretical details from Goldman and Kaplan (2018). It may provide a deeper understanding for some readers, although it may also be skipped Notationally, let F X (•) be the population CDF for the first group, and let F Y (•) be the population CDF for the second group.…”

Bond Losses and Systemic Risk

“…14 The CDB, EIB, and ADB also use a similar method to build their primary groups, but they do not impose a bidding minimum volume for primary dealers. 15 In this study, we refer to all prequalified dealers as "primary" dealers. 16 One of the most distinctive characteristics of primary dealers in China is their overlapping nature across the four bond-issuing institutions.…”