Patterns of modern human population structure are helpful in understanding the history of human migration and admixture. We conducted a study on genetic structure of the Malay population in Malaysia, using 54,794 genome-wide single nucleotide polymorphism genotype data generated in four Malay sub-ethnic groups in peninsular Malaysia (Melayu Kelantan, Melayu Minang, Melayu Jawa and Melayu Bugis). To the best of our knowledge this is the first study conducted on these four Malay sub-ethnic groups and the analysis of genotype data of these four groups were compiled together with 11 other populations' genotype data from Indonesia, China, India, Africa and indigenous populations in Peninsular Malaysia obtained from the Pan-Asian SNP database. The phylogeny of populations showed that all of the four Malay sub-ethnic groups are separated into at least three different clusters. The Melayu Jawa, Melayu Bugis and Melayu Minang have a very close genetic relationship with Indonesian populations indicating a common ancestral history, while the Melayu Kelantan formed a distinct group on the tree indicating that they are genetically different from the other Malay sub-ethnic groups. We have detected genetic structuring among the Malay populations and this could possibly be accounted for by their different historical origins. Our results provide information of the genetic differentiation between these populations and a valuable insight into the origins of the Malay sub-ethnic groups in Peninsular Malaysia.

Peninsular Malaysia is a strategic region which might have played an important role in the initial peopling and subsequent human migrations in Asia. However, the genetic diversity and history of human populations--especially indigenous populations--inhabiting this area remain poorly understood. Here, we conducted a genome-wide study using over 900,000 single nucleotide polymorphisms (SNPs) in four major Malaysian ethnic groups (MEGs; Malay, Proto-Malay, Senoi and Negrito), and made comparisons of 17 world-wide populations. Our data revealed that Peninsular Malaysia has greater genetic diversity corresponding to its role as a contact zone of both early and recent human migrations in Asia. However, each single Orang Asli (indigenous) group was less diverse with a smaller effective population size (N(e)) than a European or an East Asian population, indicating a substantial isolation of some duration for these groups. All four MEGs were genetically more similar to Asian populations than to other continental groups, and the divergence time between MEGs and East Asian populations (12,000--6,000 years ago) was also much shorter than that between East Asians and Europeans. Thus, Malaysian Orang Asli groups, despite their significantly different features, may share a common origin with the other Asian groups. Nevertheless, we identified traces of recent gene flow from non-Asians to MEGs. Finally, natural selection signatures were detected in a batch of genes associated with immune response, human height, skin pigmentation, hair and facial morphology and blood pressure in MEGs. Notable examples include SYN3 which is associated with human height in all Orang Asli groups, a height-related gene (PNPT1) and two blood pressure-related genes (CDH13 and PAX5) in Negritos. We conclude that a long isolation period, subsequent gene flow and local adaptations have jointly shaped the genetic architectures of MEGs, and this study provides insight into the peopling and human migration history in Southeast Asia.

Background: The Malays consist of various sub-ethnic groups which are believed to have different ancestral origins based on their migrations centuries ago. The sub-ethnic groups can be divided based on the region they inhabit; the northern (Melayu Kedah and Melayu Kelantan), western (Melayu Minang) and southern parts (Melayu Bugis and Melayu Jawa) of Peninsular Malaysia. We analyzed 54,794 autosomal single nucleotide polymorphisms (SNPs) which were shared by 472 unrelated individuals from 17 populations to determine the genetic structure and distributions of the ancestral genetic components in five Malay sub-ethnic groups namely Melayu Bugis, Melayu Jawa, Melayu Minang, Melayu Kedah, and Melayu Kelantan. We also have included in the analysis 12 other study populations from Thailand, Indonesia, China, India, Africa and Orang Asli sub-groups in Malay Peninsula, obtained from the Pan Asian SNP Initiative (PASNPI) Consortium and International HapMap project database. Results: We found evidence of genetic influx from Indians to Malays, more in Melayu Kedah and Melayu Kelantan which are genetically different from the other Malay sub-ethnic groups, but similar to Thai Pattani. More than 98% of these northern Malays haplotypes could be found in either Indians or Chinese populations, indicating a highly admixture pattern among populations. Nevertheless, the ancestry lines of Malays, Indonesians and Thais were traced back to have shared a common ancestor with the Proto-Malays and Chinese.

Fine scale population structure of Malays - the major population in Malaysia, has not been well studied. This may have important implications for both evolutionary and medical studies. Here, we investigated the population sub-structure of Malay involving 431 samples collected from all states from peninsular Malaysia and Singapore. We identified two major clusters of individuals corresponding to the north and south peninsular Malaysia. On an even finer scale, the genetic coordinates of the geographical Malay populations are in correlation with the latitudes (R2 = 0.3925; P = 0.029). This finding is further supported by the pairwise FST of Malay sub-populations, of which the north and south regions showed the highest differentiation (FST [North–south] = 0.0011). The collective findings therefore suggest that population sub-structure of Malays are more heterogenous than previously expected even within a small geographical region, possibly due to factors like different genetic origins, geographical isolation, could result in spurious association as demonstrated in our analysis. We suggest that cautions should be taken during the stage of study design or interpreting the association signals in disease mapping studies which are expected to be conducted in Malay population in the near future.Electronic supplementary materialThe online version of this article (doi:10.1186/s40246-015-0039-x) contains supplementary material, which is available to authorized users.