Population genetic assessment is crucial for the conservation and management of threatened species. Xanthocyparis vietnamensis is an endangered species that is currently restricted to karst mountains in southwestern China and Vietnam. This rare conifer was first recorded in 2002 from northern Vietnam and then in 2013 from Guangxi, China, yet nothing is known about its genetic diversity nor ploidy level variation, although previous cytological study suggest that Vietnamese populations are tetraploids. There have been about 45 individuals found to date in Guangxi, China. Here, we genotyped 33 X. vietnamensis individuals using 20 newly developed, polymorphic microsatellite loci, to assess the genetic variability of its extremely small populations. The genetic diversity of X. vietnamensis (HE = 0.511) was lower than that of two other heliophile species, Calocedrus macrolepis and Fokienia hodginsii, which have similar distribution ranges. This is consistent with the signature of a genetic bottleneck detected in X. vietnamensis. Although the population genetic differentiation coefficient across loci is moderate (FST = 0.125), STRUCTURE analysis revealed two distinct genetic clusters, namely the northern and southern population groups; DAPC analysis grouped the southern populations together in one cluster separate from the northern populations; AMOVA analysis detected a significant genetic differentiation between the two population groups (FRT = 0.089, p < 0.05), and BARRIER analysis detected a genetic barrier between them. Moreover, we detected differentiation in ploidy level between northern and southern populations, sampled individuals from the former and the later are all diploid and tetraploid cytotypes with mean genome sizes of 26.08 and 48.02 pg/2C, respectively. We deduced that heterogeneous geomorphology and historical events (e.g., human deforestation, Quaternary climate oscillations) may have contributed to population fragmentation and small population size in X. vietnamensis. Considering both genetic and ploidy level differentiation, we propose that two different management units (northern and southern) should be considered and a combination of in situ and ex situ conservation measures should be employed to preserve populations of this endangered species in southwestern China in the light of our findings.