Identifying conservation units is crucial for the effective conservation of threatened species. Previous cases are almost exclusively based on large‐scale but coarse sampling for genetic structure analyses. Significant genetic structure can occur within a small range, and thus multiple conservation units may exist in narrowly distributed plants. However, small‐scale genetic structure is often overlooked in conservation planning especially for wind‐pollinated and wind‐dispersed trees, largely due to the absence of dense and elaborate sampling. In this study, we focused on a representative endangered relict plant, Metasequoia glyptostroboides. Using both nuclear microsatellites (nSSRs) and chloroplast DNA (cpDNA) fragments, we sampled across the narrow distribution range of this species and determined its conservation units by exploring its genetic structure and historical demography. cpDNA haplotypes were classified into two groups, but mixed in space, suggesting that the existent wild trees of M. glyptostroboides cannot be divided into different evolutionarily significant units. However, using nSSRs, we detected strong spatial genetic structure, with significant genetic differentiation and weak gene flow between the samples in the east of the species' distribution range and other samples. The divergence between the two nSSR groups was dated to the Last Glacial Maximum (c. 19.6 kya), suggesting that such spatial genetic structure has been maintained for a long term. Therefore, these two nSSR groups should be considered as different conservation units, that is, management units, to protect intergroup genetic variations, which is likely to be the outputs of local adaptation. Our findings highlight the necessity to reveal small‐scale genetic structure and population demography to improve the conservation strategies of evolutionary potential of endangered plants.