Planting alfalfa in grey desert soil can have significant effects on soil nutrient levels, microbial communities, and overall soil improvement. High-throughput sequencing technology was used to explore the relationship between the rhizosphere microbial community structure of grey desert soil planted with different alfalfa varieties (Aohan, WL525HQ, Knight2, Kangsai, Victoria, and WL712), alfalfa characteristics and rhizosphere soil physicochemical properties. Alfalfa planting increased the nitrogen and organic matter in the grey desert soil, and the effects in Victoria, Kangsai, and Aohan were relatively better than those in the unplanted areas and other alfalfa areas. The Chao1 and Shannon indexes showed that the diversity and relative abundance of bacteria and fungi in Kangsai were significantly higher than those in the unplanted areas and other alfalfa areas. Redundancy analysis showed that available nitrogen and phosphorus, as well as fresh weight, significantly affected the changes in fungal and bacterial communities. Variance partitioning analysis showed that soil and alfalfa growth characteristics explained 50.04% and 51.58% of the structural changes in the bacteria and fungi, respectively. Therefore, planting alfalfa changed the community structure of bacteria and fungi, as well as the content of soil nutrients, and different varieties of alfalfa had different effects on soil improvement.