Vegetation plays a key role in water conservation in the southern Qilian Mountains (northwestern China), located in the upper reaches of the Heihe River. Nitrogen-fixing bacteria are crucial for the protection of the nitrogen supply for vegetation in the region. In the present study, nifH gene clone libraries were established to determine differences between the nitrogen-fixing bacterial communities of the Potentilla parvifolia shrubland and the Carex alrofusca meadow in the southern Qilian Mountains. All of the identified nitrogen-fixing bacterial clones belonged to the Proteobacteria. At the genus level, Azospirillum was only detected in the shrubland soil, while Thiocapsa, Derxia, Ectothiorhodospira, Mesorhizobium, Klebsiella, Ensifer, Methylocella and Pseudomonas were only detected in the meadow soil. The phylogenetic tree was divided into five lineages: lineages I, II and III mainly contained nifH sequences obtained from the meadow soils, while lineage IV was mainly composed of nifH sequences obtained from the shrubland soils. The Shannon–Wiener index of the nifH genes ranged from 1.5 to 2.8 and was higher in the meadow soils than in the shrubland soils. Based on these analyses of diversity and phylogeny, the plant species were hypothesised to influence N cycling by enhancing the fitness of certain nitrogen-fixing taxa. The number of nifH gene copies and colony-forming units (CFUs) of the cultured nitrogen-fixing bacteria were lower in the meadow soils than in the shrubland soils, ranging from 0.4 × 107 to 6.9 × 107 copies g−1 soil and 0.97 × 106 to 12.78 × 106 g−1 soil, respectively. Redundancy analysis (RDA) revealed that the diversity and number of the nifH gene copies were primarily correlated with aboveground biomass in the shrubland soil. In the meadow soil, nifH gene diversity was most affected by altitude, while copy number was most impacted by soil-available K. These results suggest that the nitrogen-fixing bacterial communities beneath Potentilla were different from those beneath Carex