Soil microbes play an integral role in agricultural production and soil fertility, while limited information is available about the effects of long‐term fertilization on rice (Oryza sativa L.)–rapeseed (Brassica napus L.) rotation yield, soil properties, and microbial communities in central China. Here, we report the response of plant–soil system to different mineral fertilizer inputs, including the balanced fertilization with N, P, and K (NPK) fertilizers and unbalanced fertilizers without one of the major nutrients (PK, NK, and NP) for 10 years. Compared with the NPK treatment, the average annual decrease rates of rice and rapeseed yield in the PK, NK, and NP treatments were 12.5%, 18.1%, 6.4% and 33.1%, 54.8%, 16.0%, respectively. Long‐term fertilization significantly changed the soil chemical properties, especially the pH and available K content. Compared with the NPK fertilization, the NK treatment increased the soil microbial composition and diversity, whereas PK and NP treatments had a reverse impact. Available K, pH, and slowly available K were closely related to alterations of the dominant bacterial community. On the contrary, the Olsen‐P, slowly available K, and pH were significantly correlated with the fungal community. Soil properties changed by fertilization had a more direct effect on crop yield than the microbial community; moreover, the bacterial group had a higher influence than the fungal group on crop yield. Therefore, a balanced nutrient input using NPK fertilization is an effective management strategy for soil biological fertility and sustainable crop yield.