Rational weed control in arable land is a key contributor to food security and sustainable agricultural development. However, little is known about how weed control practices affect the root-associated arbuscular mycorrhizal fungi (AMF), especially in rice-wheat cropping systems. Here, the effects of different weed control measures on AMF community structure, wheat yield and soil properties were determined based on a long-term weed control experiment established since 2001 in a rice-wheat annual double cropping system. The weed control treatments include: (ⅰ) control (CK): No weeding; (ⅱ) manual weeding + herbicide application (MH); (ⅲ) Hindering seeds dispersed from water + herbicide application (WaH); (ⅳ): Hindering seeds dispersed from water + manual weeding (WaM); (ⅴ) Hindering seeds dispersed from water + manual weeding + herbicide application (WaMH). Results demonstrated that weed control markedly affected soil properties, shifted AMF community structure (diversity and composition), and notably increased wheat yield by 52.6%-62.2% compared with CK. The values of Pielou evenness index, Shannon-Weiner index and Simpson’s index were significantly lower under MH compared with CK. The Mantel test analysis indicated that AMF diversity was dominantly controlled by both total nitrogen and soil organic carbon. Similarly, AMF composition was markedly affected by soil water content, total nitrogen, nitrate nitrogen, ammonium nitrogen, and microbial biomass nitrogen. Mantel test also revealed that wheat yield was significantly correlated to AMF composition and Pearson correlation analysis also showed that some key indicator taxa were significantly correlated with wheat yield, suggesting that the composition of AMF community might play an important role in wheat yield formation. In addition, redundancy analysis (RDA) showed that soil properties and wheat yield were closely related to AMF community characteristics. Taken together, our findings suggest that weed control can mediate the relationships among soil quality, AMF community and wheat yield, providing valuable guidance for integrated weed management in the Yangtze River basin regions in China.