Error Rates in Multiple Comparisons Among Means-Results of a Simulation Exercise

Abstract

Summary When a number of tests are carried out simultaneously on a set of data some overall guarantee against making type I errors is required. Various solutions to this problem have been suggested with different approaches to this overall protection level. The joint operating characteristics of simultaneous tests are usually incompletely specified and simulation has to be employed to evaluate the power of the tests. A new simulation is reported, comparing seven methods of pairwise comparisons and four for constructing simultaneous sets of confidence limits. The general conclusions are that Duncan's multiple range test is the best method of those considered for the former and the Bonferroni t‐based limits for the latter. The findings are generally in agreement with all but one of the previous simulations which have been carried out on multiple comparisons.

A comparison of mean separation techniques

“…In this paper, a* (the experiment-wise error rate) is defined for a given set of means (just as power is defined for given means), and, by the reasoning given just prior to the Methods section, the prior F test will cause a* to decrease. Thomas (1974) has done a simulation with n =4, oZ = 1, k = 5, 10 and 20 populations, L = 1,2, and k levels, and Cj = 4 or 6, with 100 to 400 replications per set. Also minimum and maximum situations were performed when L = 2.…”

The Epistemology of Statistical Science

“…The demand has resulted in a stream of increasingly obscure procedures, so much so that Monte Carlo studies have been conducted to try to fathom what the methods actually do. Chew (1976) for instance describes four Monte Carlo studies (by Boardman and Moffit, 1971;Carmer and Swanson, 1973;Thomas, 1974;and Einot and Gabriel, 1975), but he seems to have been unable to make much use of them. It cannot but be considered very odd that a community of mathematically talent individuals have devised many procedures, persuaded thousands of investigators in substantive science to use those procedures and, having done that, have then resorted to simulation studies in the hope of finding out what they have wrought by way of the procedures.…”

Statistical power of multiplicity adjustment strategies for correlated binary endpoints

“…In this context, with alpha thresholds that vary, the Holm and Hochberg procedures can each provide more statistical power than a strict application of the Bonferroni adjustment. The statistical power of strategies for multiple tests of group means has been evaluated [6][7][8][9][10][11]. Yet, none of the aforementioned approaches incorporates the correlations between endpoints.…”