Kruskal–Wallis, Multiple Comparisons and Efron Dice

Abstract

The Kruskal-Wallis test is a rank-based one way ANOVA. Its test statistic is shown here to be a quadratic form among the Mann-Whitney or Kendall tau concordance measures between pairs of treatments. But the full set of such concordance measures has more degrees of freedom than the Kruskal-Wallis test uses, and the independent surplus is attributable to circularity, or non-transitive effects. The meaning of circularity is well illustrated by Efron dice. The cases of k = 3, 4 treatments are analysed thoroughly in this paper, which also shows how the full sum of squares among all concordance measures can be decomposed into uncorrelated transitive and non-transitive circularity effects. A multiple comparisons procedure based on patterns of transitive orderings among treatments is implemented. The testing of circularities involves non-standard asymptotic distributions. The asymptotic theory is deferred, but Monte Carlo permutation tests are easy to implement.

A Regression Framework for Rank Tests Based on the Probabilistic Index Model

“…In this section we establish the relationship between the pairwise PIM and the rank test of Brown and Hettmansperger (2002) for the three-sample layout. Brown and Hettmansperger (2002) showed that the KW test cannot test for intransitivity and therefore they proposed to extend the KW test statistic as follows…”

“…Brown and Hettmansperger (2002) showed that the KW test cannot test for intransitivity and therefore they proposed to extend the KW test statistic as follows…”

“…The full connection between a PIM and rank tests were not exploited in Thas et al (2012) since specific parametrizations are required. More specifically, depending on this parametrization, we establish a simple connection between the PIM and the WMW, KW, Friedman (Friedman, 1937), Mack-Skillings (MS) (Mack and Skillings, 1980), Brown-Hettmansperger (BH) (Brown and Hettmansperger, 2002) and the Jonckheere-Terpstra (JT) (Jonckheere, 1954;Terpstra, 1952) rank tests and some tests proposed by Brunner and Puri (2001) based on the work of Akritas and Arnold (1994).…”

How healthy are software engineering conferences?

“…For a particular metric, nodes of the graph correspond to conferences, and edges to results of pairwise comparisons (there is an edge from A to B if A tends to have higher values for that metric than B). Because transitivity is respected by T (as opposed to, e.g., the traditional pairwise Wilcoxon-Mann-Whitney tests [40]), we omit direct edges between A and B if there is a path from A to B passing through at least one other node C.…”

The two-step approach—a significant ANOVA F-test before Dunnett's comparisons against a control—is not recommended

“…There exist different formulations for both all-pairs comparisons and many-to-one comparisons of nonparametric tests, such as the all-pairs rank test Steel (1959) and the manyto-one rank test (Steel, 1959) which differ in their power. Moreover, the asymptotic all-pairs k-ranking approaches (Nemenyi, 1963;Dunn, 1964) do not control the family-wise error rate (Brown and Hettmansperger, 2002), which is particularly of interest with variance heterogeneity. The small sample behavior of all tests is quite different for their asymptotic, simulationbased or exact test versions (e.g., see Hothorn et al, 2006).…”