Power comparisons of significance tests of location using scores, ranks, and modular ranks

Abstract

The Type I error probability and the power of the independent samples t test, performed directly on the ranks of scores in combined samples in place of the original scores, are known to be the same as those of the non-parametric Wilcoxon-Mann-Whitney (WMW) test. In the present study, simulations revealed that these probabilities remain essentially unchanged when the number of ranks is reduced by assigning the same rank to multiple ordered scores. For example, if 200 ranks are reduced to as few as 20, or 10, or 5 ranks by replacing sequences of consecutive ranks by a single number, the Type I error probability and power stay about the same. Significance tests performed on these modular ranks consistently reproduce familiar findings about the comparative power of the t test and the WMW tests for normal and various non-normal distributions. Similar results are obtained for modular ranks used in comparing the one-sample t test and the Wilcoxon signed ranks test.

Are Assumptions of Well-Known Statistical Techniques Checked, and Why (Not)?

“…Furthermore, many participants seemed familiar with a rule of thumb to check whether the assumption of homogeneity of variance for ANOVA is violated (e.g., largest standard deviation is larger than twice the smallest standard deviation), whereas such rules of thumb for checking possible violations of the assumption of normality were unknown to our participants. It was also found that Levene’s test was often used as a preliminary test to choose between the pooled t- test and the Welch t- test, despite the fact that the use of preliminary tests is often discouraged (e.g., Wilcox et al, 1986; Zimmerman, 2004, 2011; Schucany and Ng, 2006). An obvious explanation for this could be that the outcomes of Levene’s test are given as a default option for the t procedure in SPSS (this was the case in all versions that were used by the participants).…”

“…Preliminary tests could, for example, be used to choose between a pooled t- test and a Welch t- test or between ANOVA and a non-parametric alternative. Following the argument that preliminary tests should not be used because, amongst others, they can inflate the probability of making a Type I error (e.g., Gans, 1981; Wilcox et al, 1986; Best and Rayner, 1987; Zimmerman, 2004, 2011; Schoder et al, 2006; Schucany and Ng, 2006; Rochon and Kieser, 2011), it has also been argued that in many cases unconditional techniques should be the techniques of choice (Hayes and Cai, 2007). For example, the Welch t- test, which does not require homogeneity of variance, would be seen a priori as preferable to the pooled variance t- test (Zimmerman, 1996; Hayes and Cai, 2007).…”

A Randomized Controlled Trial of a Standardized Behavior Management Intervention for Students With Externalizing Behavior

“…The distributions of the observer ratings of student externalizing behavior and teacher reprimands were skewed. Therefore, those variables were transformed using modular ranks before further analyses were conducted (Zimmerman, 1992, in press). The variance homogeneity of the nontransformed variables was tested with Levene’s test, since heterogeneity has been found to inflate the risk of Type I error in rank transformations (Zimmerman, 2004).…”

Statistical Conclusion Validity: Some Common Threats and Simple Remedies

“…A second solution consists of using classical test statistics that have been shown to be robust to violation of their assumptions. Indeed, dependable unconditional tests for means or for regression parameters have been identified (see Sullivan and D’Agostino, 1992; Lumley et al, 2002; Zimmerman, 2004, 2011; Hayes and Cai, 2007; Ng and Wilcox, 2011). And a third solution is switching to modern robust methods (see, e.g., Wilcox and Keselman, 2003; Keselman et al, 2004; Wilcox, 2006; Erceg-Hurn and Mirosevich, 2008; Fried and Dehling, 2011).…”