New machine learning method identifies subtle fine-scale genetic stratification in diverse populations

Abstract

Fine-scale genetic structure impacts genetic risk predictions and furthers the understanding of the demography of diverse population which may lead to health disparities in population health monitoring system. Epidemiology and population genetic research try to disentangle subtle genetic differences within a population through various dimension reduction approaches (i.e., PCA, DAPC, t-SNE, and UMAP). However, one undesirable aspect of these approaches is that they either produce coarse and ambiguous cluster divisions or they fail to preserve the correct genetic distance between populations. We proposed a new machine learning algorithm (ALFDA) for better capturing fine-scale genetic structure and recapitulating geogenetic distance. ALFDA correctly estimated the genetic affinity between individuals and keep the multimodal structure within populations. Through extensive simulations and empirical data analyses, we showed that ALFDA outperformed the other approaches in identifying fine-scale genetic structure and in preserving geogenetic distance. Notably, genetic features produced from ALFDA had highest correlation with F_ST under an isolation-by-distance model. We identified a rich pattern of subtle fine-scale genetic differentiation within diverse populations than those identified by existing approaches, indicating that genetic ancestry is more nuanced than previously reported. Our method was able to uncover fine-scale genetic structure within populations, which could help understand the genetic origin and diversity of individuals in understudied diverse populations thus facilitates precise health.