The microbial effect of long-term heavy metal pollution on farmland remains unclear. Here, we investigated microbial (bacterial and fungal) communities in topsoil and rhizosphere samples with heavy metal (Cd, Cu, Pb, and Zn) pollution from four different types of tillage plots around an abandoned zinc smelter set up 85 years ago and analyzed the complex relationship between microorganisms, plants, and heavy metals (HMs) in soil to guide strategies for further soil remediation measures. The abundance and diversity index results showed that the bacterial and fungal diversities of the four plots were significantly different. Meanwhile, correlation analysis of the microbial communities and HMs showed that bacteria Pseudomonas and fungi Chaetothyriales and Fusarium had a good tolerance for HM pollution, but bacteria Vicinamibacteraceae, JG30_KF_CM45, RB41, Gaiella, MB-A2-108, 67-14, and Microvirga, and fungi Glomerellales, Hypocreales, Chaetomium, and Mortierella all showed indications of being sensitive to HM toxicity. Our structural equation model (SEM) attributed the inhibition of Zn and the promotion of Cd to bacterial diversity, attributed a weak inhibition of Cd to fungal diversity, and revealed the effects of the tillage type on these diversities.