A comparison of multiple testing adjustment methods with block-correlation positively-dependent tests

Abstract

In high dimensional data analysis (such as gene expression, spatial epidemiology, or brain imaging studies), we often test thousands or more hypotheses simultaneously. As the number of tests increases, the chance of observing some statistically significant tests is very high even when all null hypotheses are true. Consequently, we could reach incorrect conclusions regarding the hypotheses. Researchers frequently use multiplicity adjustment methods to control type I error rates—primarily the family-wise error rate (FWER) or the false discovery rate (FDR)—while still desiring high statistical power. In practice, such studies may have dependent test statistics (or p-values) as tests can be dependent on each other. However, some commonly-used multiplicity adjustment methods assume independent tests. We perform a simulation study comparing several of the most common adjustment methods involved in multiple hypothesis testing, under varying degrees of block-correlation positive dependence among tests.

Can't Ignore—Distraction by Task‐Irrelevant Sounds in Early and Middle Childhood

“…Post hoc tests were performed by recoding of the reference category until all 10 pairwise comparisons of the distraction effects between the age groups were conducted. We controlled the false discovery rate (FDR) at a level of 5% using the two-step Benjamini-Hochberg procedure (Benjamini, Krieger, & Yekutieli, 2006), because it provides a good compromise between statistical power and FDR control (Stevens, Al Masud, & Suyundikov, 2017). All LMMs were conducted using the packages lme4 (Bates, Maechler, Bolker, & Walker, 2015) and lmerTest (Kuznetsova, Brockhoff, Haubo, & Christensen, 2016) in R. In order to maximize precision of random-effect estimates, the models were estimated using restricted maximum likelihood where appropriate.…”

The influence of enamel sandblasting on the shear bond strength and fractography of the bracket-adhesive-enamel complex tested in vitro by the DIN 13990:2017-04 standard

“…We conducted descriptive and inferential statistics using IBM SPSS Statistics 23 (IBM Corporation, New York, USA). The dependence of shear strength from the enamel preparation protocol was analysed by oneway analysis of variance (ANOVA) after checking for normality of residuals and homogeneity of variances (α=5%) followed by post hoc pairwise comparisons with Sidak p-value correction [36]. We analysed the fractography patterns resulting from the enamel preparation protocols applying Fisher's exact test, as 50-75 % of the expected frequencies were less than five.…”

Mapping co-ancestry connections between the genome of a Medieval individual and modern Europeans

“…However, the large number of tests performed with the Medieval are likely to increase type I error rates (also the binomial tests are not independent because the finding of IBD between the Medieval and a given population is likely to influence subsequent tests) 58 . In similar cases of block-positive dependence among tests, it has been shown that a best option to control for false discovery rate (FDR) 59 is to use the two stage Benjamini-Hochberg (TSBH) procedure 60 . We subsequently adjusted the p-values between observed and expected IBD blocks for the TSBH procedure; a nominal type I error rate (5%) was used to estimate the number of true null hypotheses in the two-stage TSBH with R multtest package 61 .…”