Elymus breviaristatus is a grass species only distributed in the southeast of Qinghai-Tibetan Plateau (QTP), which has suffered from serious habitat fragmentation. Therefore, understanding patterns of genetic diversity within and among natural E. breviaristatus populations could provide insight for future conservation strategies. In this study, sequence-related amplified polymorphism markers were employed to investigate the genetic diversity and hierarchical structure of seven E. breviaristatus populations from QTP, China. Multiple measures of genetic diversity indicated that there is low to moderate genetic variation within E. breviaristatus populations, consistent with its presumed mating system. In spite of its rarity, E. breviaristatus presented high genetic diversity that was equivalent to or even higher than that of widespread species. Bayesian clustering approaches, along with clustering analysis and principal coordinate analysis partitioned the studied populations of E. breviaristatus into five genetic clusters. Differentiation coefficients (Fst, GST, etc.) and AMOVA analysis revealed considerable genetic divergence among different populations. BARRIER analyses indicated that there were two potential barriers to gene flow among the E. breviaristatus populations. Despite these patterns of differentiation, genetic distances between populations were independent of geographic distances (r = 0.2197, p = 0.2534), indicating little isolation by distance. Moreover, despite detecting a common outlier by two methods, bioclimatic factors (altitude, annual mean temperature, and annual mean precipitation) were not related to diversity parameters, indicating little evidence for isolation caused by the environment. These patterns of diversity within and between populations are used to propose a conservation strategy for E. breviaristatus.