Vegetation plays a key role to water conservation in southern Qilian Mountains (Northwestern China), the upper reaches of Heihe River. Nitrogen-fixing bacteria are crucial for vegetation protection because they can supply plants with nitrogen source. Nevertheless, little is known about nitrogen-fixing bacteria in this region. In present study, nifH gene clone libraries were established for detecting the difference of nitrogen-fixing bacterial communities between Potentilla parvifolia shrub and Carex alrofusca meadow in the southern Qilian Mountains. All the identified nitrogen-fixing bacterial clones belonged to Proteobacteria. At the genus level, the Azospirillum sp. was only detected in shrub soil while Thiocapsa sp., Derxiasp., Ectothiorhodospira sp., Mesorhizobium sp., Klebsiella sp., Ensifer sp., Methylocella sp. and Peseudomonas sp. were just detected in meadow soil. Shannon–Wiener index of nifH gene ranged from 1.5 to 2.8 and was higher in meadow soil than shrub soil. Contrarily, the nifH gene copies and CFUs of cultured nitrogen-fixing bacteria ranged from 0.4 × 107 to 6.9 × 107 copies g−1 soil and 0.97 × 106 to 12.78 × 106 g−1 soil, respectively. Furthermore, both of them were lower in meadow soil than shrub soil. Statistical analysis revealed that diversity and copies of nifH gene mostly correlated with aboveground biomass in shrub soil. In meadow soil, nifH gene diversity was principally affected by altitude while copies did by soil available K