Grazing has been shown to impact the soil environment and microbial necromass carbon (MNC), which in turn regulates soil organic carbon (SOC). However, the carbon sequestration potential of fungi and bacteria under different stocking rates remains unclear, limiting our understanding of soil carbon sequestration in grazing management. In 2004, we established grazing experiments in the desert steppe of northern China with four stocking rates. Our findings indicate that MNC decreased under moderate and heavy grazing, while light grazing did not significantly differ from no grazing. Notably, the reduction in fungal necromass carbon, rather than bacterial necromass carbon, was primarily responsible for the decreased contribution of MNC to SOC. This difference is attributed to the varying effects of sheep grazing on fungal and bacterial community characteristics, including richness, diversity, and composition. Thus, to accurately predict carbon dynamics in grassland ecosystems, it is essential to consider that the ecological impacts and carbon sequestration potential of microbial communities may vary with different grazing management practices.