Beyond the traditional simulation design for evaluating type 1 error control: From the “theoretical” null to “empirical” null

Zhang, Ting; Sun, Lei

doi:10.1002/gepi.22172

Cited by 5 publications

(6 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our simulations, we have considered various types of model misspecification and demonstrated robustness of BRASS for association testing. We note that certain applications such as testing of interaction have been shown to be more sensitive to model misspecification than association testing is [25], so additional sensitivity analyses may be needed when performing interaction analyses, regardless of the method used to assess significance.…”

Section: Discussionmentioning

confidence: 99%

BRASS: permutation methods for binary traits in genetic association studies with structured samples

Mbatchou

Abney

McPeek

2018

Preprint

View full text Add to dashboard Cite

In genetic association analysis of complex traits, permutation testing can be a valuable tool for assessing significance when the distribution of the test statistic is unknown or not well-approximated. This commonly arises when the association test statistic is itself a function of multiple correlated statistics, e.g, in tests of gene-set, pathway or genome-wide significance, as well as omnibus tests that combine test statistics that perform well in different scenarios. For genetic association testing in samples with population structure and/or relatedness, use of naive permutation can lead to inflated type 1 error. To address this in quantitative traits, the MVNpermute method was developed. However, for association mapping of a binary trait, the relationship between the mean and variance makes both naive permutation and the MVNpermute method invalid. We propose BRASS, a permutation method for binary trait association mapping in samples that have related individuals and/or population structure. BRASS allows for covariates, ascertainment and simultaneous testing of multiple markers, and it accommodates a wide range of test statistics. We use an estimating equation approach that can be viewed as a hybrid of logistic regression and linear mixed-effects model methods, and we use a combination of principal components and a genetic relatedness matrix to account for sample structure. In simulation studies, we compare BRASS to other permutation and resampling-based methods in a range of scenarios that include population structure, familial relatedness, ascertainment and phenotype model misspecification. In these settings, only BRASS maintains correct control of type 1 error, performing far better than all other methods. We apply BRASS to two genome-wide analyses in domestic dog, one for elbow dysplasia (ED) in 82 breeds and another for idiopathic epilepsy (IE) in the Irish Wolfhound breed. We detect significant association of IE with SNPs in a previously-identified chromosome 4 region that contains multiple candidate genes. Author summaryPermutation testing is commonly used when distributional assumptions cannot be made or do not apply, or when performing a multiple testing correction, e.g., to assess region-wide or genome-wide significance in association mapping studies. Naively permuting the data is only valid under the assumption of exchangeability, which, in the presence of sample structure and polygenicity, typically does not hold. Linear mixed-model based approaches have been proposed for permutation-based tests with continuous traits that can also adjust for sample structure; however, these may not February 5, 2019 1/28 remain valid when applied to binary traits, as key features of binary data are not well accounted for. We propose BRASS, a permutation-based testing method for binary data that incorporates important characteristics of binary data in the trait model, can accommodate relevant covariates and ascertainment, and adjusts for the presence of structure in the sample. We demonstrate the use of this approac...

show abstract

Section: Discussionmentioning

confidence: 99%

BRASS: permutation methods for binary traits in genetic association studies with structured samples

Mbatchou

Abney

McPeek

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…The second approach assumes a more general and robust version of Model (4) in which we allow a specific type of heteroscedasticity, namely, we allow to depend quadratically on Z , and we call this the heteroscedastic model. In an interaction GWAS, it can potentially be important to consider this specific type of heteroscedasticity, because it arises naturally in a model in which Z interacts with some other variable in a linear model or LMM for Y , even if it doesn’t interact with G j [39–41, 43]. That is, suppose the true model for Y could be written where Y, U, α, G j , β, Z, γ , and ϵ are as before, ζ and θ are unknown scalar coefficients, and X is some additional variable that might or might not be observed, is independent of ( G j , Z ), and that interacts with Z .…”

Section: Methodsmentioning

confidence: 99%

“…For example, in some cases, an apparent epistatic effect that is detected could be due to an unsequenced causal variant [34,37,38]. Another important issue that has been identified is heteroscedasticity [39][40][41] that can result under the null model when, for example, interaction is present between one of the two tested variables and some other variable not included in the model or when the null model is misspecified in some other way. If not accounted for, this heteroscedasticity can lead to excess type 1 error [39][40][41].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Overcoming the “feast or famine” effect: improved interaction testing in genome-wide association studies

Zhou,

McPeek

2024

Preprint

View full text Add to dashboard Cite

In genetic association analysis of complex traits, detection of interaction (either GxG or GxE) can help to elucidate the genetic architecture and biological mechanisms underlying the trait. Detection of interaction in a genome-wide association study (GWAS) can be methodologically challenging for various reasons, including a high burden of multiple comparisons when testing for epistasis between all possible pairs of a set of genomewide variants, as well as heteroscedasticity effects occurring in the presence of GxG or GxE interaction. In this paper, we address the problem of an even more striking phenomenon that we call the "feast or famine" effect that occurs when testing interaction in a genomewide context. As we verify, even in a simplified setting in which there is no interaction at all (and so no heteroscedasticity), in a GWAS to detect GxG or GxE interaction with a fixed genetic variant or environmental factor, the distribution of the genome-wide p-values under the null hypothesis is not the i.i.d. uniform one that is commonly assumed. Using standard methods, even if all SNPs are independent, some GWASs will have systematically underinflated p-values ("feast"), and others will have systematically overinflated p-values ("famine"), which can lead to false detection of interaction, reduced power, inconsistent results across studies, and failure to replicate true signal. This startling phenomenon is specific to detection of interaction in a GWAS, and it may partly explain why such detection has so far proved challenging and difficult to replicate. We show theoretically that the key cause of this phenomenon is which variables are conditioned on in the analysis, and this suggests an approach to correct the problem by changing the way the conditioning is done. Using this insight, we have developed the TINGA method to adjust the interaction test statistics to make their p-values closer to uniform under the null hypothesis. In simulations we show that TINGA both controls type 1 error and improves power. TINGA allows for covariates and population structure through use of a linear mixed model and accounts for heterscedasticity. We apply TINGA to detection of epistasis in a study of flowering time in Arabidopsis thaliana.

show abstract

“…Genotype, trait and covariates are generated under multiple simulation settings. In real data applications, the true trait model is usually not known a priori, and it can be important to assess the impact of model misspecification on type 1 error [25]. We consider the effects of model misspecification due to (1) assuming a logistic link function when the true model is a liability threshold model and (2) exclusion of an important covariate from the model.…”

Section: Methods For Simulation Studiesmentioning

confidence: 99%

BRASS: Permutation methods for binary traits in genetic association studies with structured samples

Mbatchou,

Abney,

McPeek

2023

PLoS Genet

View full text Add to dashboard Cite

In genetic association analysis of complex traits, permutation testing can be a valuable tool for assessing significance when the distribution of the test statistic is unknown or not well-approximated. This commonly arises, e.g, in tests of gene-set, pathway or genome-wide significance, or when the statistic is formed by machine learning or data adaptive methods. Existing applications include eQTL mapping, association testing with rare variants, inclusion of admixed individuals in genetic association analysis, and epistasis detection among many others. For genetic association testing in samples with population structure and/or relatedness, use of naive permutation can lead to inflated type 1 error. To address this in quantitative traits, the MVNpermute method was developed. However, for association mapping of a binary trait, the relationship between the mean and variance makes both naive permutation and the MVNpermute method invalid. We propose BRASS, a permutation method for binary traits, for use in association mapping in structured samples. In addition to modeling structure in the sample, BRASS allows for covariates, ascertainment and simultaneous testing of multiple markers, and it accommodates a wide range of test statistics. In simulation studies, we compare BRASS to other permutation and resampling-based methods in a range of scenarios that include population structure, familial relatedness, ascertainment and phenotype model misspecification. In these settings, we demonstrate the superior control of type 1 error by BRASS compared to the other 6 methods considered. We apply BRASS to assess genome-wide significance for association analyses in domestic dog for elbow dysplasia (ED) and idiopathic epilepsy (IE). For both traits we detect previously identified associations, and in addition, for ED, we detect significant association with a SNP on chromosome 35 that was not detected by previous analyses, demonstrating the potential of the method.

show abstract

Beyond the traditional simulation design for evaluating type 1 error control: From the “theoretical” null to “empirical” null

Cited by 5 publications

References 36 publications

BRASS: permutation methods for binary traits in genetic association studies with structured samples

BRASS: permutation methods for binary traits in genetic association studies with structured samples

Overcoming the “feast or famine” effect: improved interaction testing in genome-wide association studies

BRASS: Permutation methods for binary traits in genetic association studies with structured samples

Contact Info

Product

Resources

About