Implementing false discovery rate control: increasing your power

Abstract

2005. Implementing false discovery rate control: increasing your power. Á/ Oikos 108: 643 Á/647. Popular procedures to control the chance of making type I errors when multiple statistical tests are performed come at a high cost: a reduction in power. As the number of tests increases, power for an individual test may become unacceptably low. This is a consequence of minimizing the chance of making even a single type I error, which is the aim of, for instance, the Bonferroni and sequential Bonferroni procedures. An alternative approach, control of the false discovery rate (FDR), has recently been advocated for ecological studies. This approach aims at controlling the proportion of significant results that are in fact type I errors. Keeping the proportion of type I errors low among all significant results is a sensible, powerful, and easy-to-interpret way of addressing the multiple testing issue. To encourage practical use of the approach, in this note we illustrate how the proposed procedure works, we compare it to more traditional methods that control the familywise error rate, and we discuss some recent useful developments in FDR control.The appropriate threshold to declare a test statistic's p value significant becomes complex when more than one test is performed. In the absence of a true effect each test has a chance of a to yield a significant result, and the chance of drawing at least one false conclusion increases rapidly with the number of tests performed. Protection against false rejections of the null hypothesis, or type I errors, is usually achieved via a Bonferroni-type correction procedure (Holm 1979). By performing individual tests at error rates that are a fraction of the overall nominal a, the chance of making even a single type I error can be maintained at the desired a level (usually 5%). This is called control of the familywise error rate (FWER). With an increasing number of tests, maintaining a low chance of making even one type I error comes at the direct cost of making more type II errors, i.e. not recognizing a true effect as significant. The classical Bonferroni procedure, which performs each of m tests at a type I error rate of a/m, is undesirable because of this trade-off: only a very strong effect is likely to be recognized as significant when many tests are performed. Several improvements to the classical Bonferroni have been proposed in order to reduce the problem of low power (reviewed by García 2004). For instance, the well known Holm's step-down or sequential Bonferroni procedure (Holm 1979, popularized among evolutionary biologists and ecologists by Rice 1989) performs tests in order of increasing p values, and conditional on having rejected tests with smaller p values an increasingly permissive threshold can be used while maintaining the FWER at the desired level (5%). Further power gains are possible with the sequential approach by using a step-up instead of a step-down procedure (that is, testing in order of decreasing p values, Hochberg 1988), by estimating the number of true nu...

Using genetic methods to investigate dispersal in two badger (Meles meles) populations with different ecological characteristics

“…Populations were tested for linkage disequilibria among loci using an exact test based on a Markov-chain method as implemented in GENEPOP 3.4. The false discovery rate technique was used to eliminate false assignment of significance by chance (Verhoeven et al, 2005). The F IS values (Weir and Cockerham, 1984) were calculated for each population using SPAGEDI 1.2 (Hardy and Vekemans, 2002) and significance was tested with 10 000 permutations of individual genotypes within populations.…”

Strong effect of climate on ectomycorrhizal fungal composition: evidence from range overlap between two mountains

“…The mountain-shared species were tested to determine whether the frequency of the species was biased toward particular forest types using Fisher's exact test. The Benjamini and Hochberg false discovery rate correction (Verhoeven et al, 2005) was performed to adjust type I error for multiple comparisons using the fmsb package of R.…”

Mito-nuclear discordance in the degree of population differentiation in a marine goby

“…The significance for R ST value and dm 2 value were tested in SPAGeDi also against 10 4 random permutations. A sequential Bonferroni test (Rice, 1989) and the false discovery rate control (Verhoeven et al, 2005) were applied to correct significance levels for multiple testing. Standardized genetic differentiation measures were obtained by dividing F ST measures by the maximum values for F ST (Hedrick, 2005;Meirmans, 2006), calculated using the pragmatic recoding approach suggested by Meirmans (2006).…”