India is home to thousands of ethno-linguistically distinct groups, many maintaining strong self-identities that derive from oral traditions and histories. However, these traditions and histories are only partially documented and are in danger of being lost over time. More recently, genetic studies have established the existence of ancestry gradients derived from both western and eastern Eurasia as well as evidence of practices such as endogamy and consanguinity, revealing complexity in the regional population structure with consequences for the health landscape of local populations. Despite the increase in genome-wide data from India, there is still sparse sampling across finer-scale geographic regions leading to gaps in our understanding of how and when present-day genetic structure came into existence. To address the gaps in genetic and oral histories, we analyzed whole-genome sequences of 70 individuals from Southwest India identifying as Bunt, Kodava, and Nair -populations that share unique oral histories and origin narratives- and 78 recent immigrants to the United States with Kodava ancestry as part of a community-led initiative. We additionally generated genome-wide data from 10 individuals self-identifying as Kapla, a population from the same region that is socio-culturally different to the other three study populations. We supplemented existing but limited anthropological records on these populations with oral history accounts narrated by community members and non-member contacts during sampling and subsequent community engagement. Overall, we find that components of genetic ancestry are relatively homogeneous among the Bunt, Kodava, and Nair populations and comparable to neighboring populations in India, which motivates further investigation of non-local origin narratives referenced in their oral histories. A notable exception is the Kapla population, with a higher proportion of ancestry represented in the Onge from the Andaman Islands, similar to several South Indian tribal populations. Utilizing haplotype-based methods, we find latent genetic structure across South India, including the sampled populations from Southwest India, suggesting more recent population structure between geographically proximal populations in the region. This study represents an attempt for community-engaged anthropological and genetic investigations in India and presents results from both sources, underscoring the need to recognize that oral and genetic histories should not be expected to overlap. Ultimately, oral traditions and unique self-identities, such as those held close by some of the study populations, warrant more community-driven anthropological investigations to better understand how they originate and their relationship to genetic histories.