Multiple Comparison Procedures—Cutting the Gordian Knot

Abstract

Multiple comparison procedures (MCPs), or mean separation tests, have been the subject of great controversy since the 1950s. Essentially, these procedures are an attempt at simultaneously formulating and testing pairwise comparison hypotheses using data from a single experiment. An unacceptable operating characteristic of most MCPs is their “inconsistency,” an idea that is illustrated in this article. This characteristic led to the development of a “practical solution” to the MCP problem, which is to “cut the Gordian knot” by abandoning any attempt at simultaneous formulation and testing. Instead, I recommend using the simplest multiple comparison procedure, the unrestricted least significant difference procedure, to (i) formulate new hypotheses at a known “false discovery rate” (in the null case) such as 5%, and (ii) independently test interesting new hypotheses in a second experiment. I also discuss the implications for sample size calculations of the choice of MCP.

Milk Composition in the Weddell SealLeptonychotes weddellii: Evidence for a Functional Role of Milk Carbohydrates in Pinnipeds

“…In case of significance ( ), group differences were investigated P ! 0.05 using a conservative multiple comparison procedure, Tukey's test (Saville 1985), with significance for group differences set at . Interindividual differences in milk composition P !…”

Thou Shall Not Brush Your Teeth While Eating Breakfast: A 7- Step Program for Researchers Previously Hurt in Data Analysis

“…But more often than not, estimation of multiple treatment differences from the same body of data are involved, leading to an increased Type I error rate. There is plenty of advice in the literature on how to deal with such situations (Carmer et al, 1989;Carmer and Swanson, 1971;Milliken and Johnson, 2009;Saville, 2013;Stroup, 2013), but little general consensus beyond the realization that some kind of protocol or procedure is needed to deal with such situations.Openness is the key; tell the reviewer/reader about the procedures that have been used If the reviewer is not convinced, he/she has the opportunity to challenge assumptions or procedures.…”

Fiber composition of a diversity panel of the world collection of sugarcane (Saccharum spp.) and related grasses

“…Stapf ), were grouped together into "Other Genera" (Nayak et al 2014). The least squared means (lsmeans) of species were compared with the DIFF option in PROC MIXED in SAS (2011), and the lsmeans of individual accessions were compared against the combined average of the 10 sugarcane checks with the LSD method (p = 0.05) (Saville 2015). The coefficient of variation (CV) was calculated by using the same Proc Mixed model together and then run for each group separately with the following equation:…”