VariantSpark: Cloud-based machine learning for association study of complex phenotype and large-scale genomic data

Abstract

Background Many traits and diseases are thought to be driven by >1 gene (polygenic). Polygenic risk scores (PRS) hence expand on genome-wide association studies by taking multiple genes into account when risk models are built. However, PRS only considers the additive effect of individual genes but not epistatic interactions or the combination of individual and interacting drivers. While evidence of epistatic interactions ais found in small datasets, large datasets have not been processed yet owing to the high computational complexity of the search for epistatic interactions. Findings We have developed VariantSpark, a distributed machine learning framework able to perform association analysis for complex phenotypes that are polygenic and potentially involve a large number of epistatic interactions. Efficient multi-layer parallelization allows VariantSpark to scale to the whole genome of population-scale datasets with 100,000,000 genomic variants and 100,000 samples. Conclusions Compared with traditional monogenic genome-wide association studies, VariantSpark better identifies genomic variants associated with complex phenotypes. VariantSpark is 3.6 times faster than ReForeSt and the only method able to scale to ultra-high-dimensional genomic data in a manageable time.

Thresholding Gini Variable Importance with a single trained Random Forest: An Empirical Bayes Approach

“…See Bayat et al (2020), where we introduced VariantSpark, a tailored Hadoop/Spark-based implementation of random forests, designed to handle data with extremely large number of variables per sample (O’Brien et al, 2018). In a test of the algorithm, VariantSpark was used to accurately predict ethnicity from whole genome sequencing profiles of 2,500 individuals and 80 million genomic variants each (1000 Genomes Project).…”

“…In conjunction with RF implementations for high-dimensional data, e.g. VariantSpark Bayat et al (2020), Random Forest can deliver on its promise of offering higher performance for GWAS-approaches than other multi-locus models (lasso and the elastic net, (Friedman et al, 2010)) and single loci mapping methods (Michaelson et al, 2010).…”

“…Lundberg et al (2019) concludes that no nonlinear model had a more widespread popularity, from health care to academia to industry, than random forests and decision trees. The popularity is in part due to the ability of RF to process large volumes of data, as demonstrated by Bayat et al (2020) who process 80 million variables.…”

“…The bounds of the methodology are still being extended. Bayat et al (2020) give an example with 80 million variables.It is highly desirable that RF models be made more interpretable and a large part of that is a better understanding of the characteristics of the variable importance measures generated by the RF. Due to its speed and ease of calculation, we consider the mean decrease in node “impurity” (MDI) variable importance (VI) and address the question of setting a significance level.The report is organized as follows: we first consider the question of multiple testing in the case of multiple measurements made on two groups (the standard microarray set up Efron (2008)).…”

“…The bounds of the methodology are still being extended. Bayat et al (2020) give an example with 80 million variables.…”