Changes in plants and soils during natural succession have been evaluated, but little is known about the effects of succession on the activities of soil microbes and their interactions with soil erodibility. We conducted a field study on the Chinese Loess Plateau, typical of this semiarid area, to determine the effect of secondary succession on the stability of soil structure against erosion and on the composition of soil fungal communities. Characteristics of plant, soil, and fungal communities were assessed across a 30-yr chronosequence of grassland developed from abandoned cropland. The diversity and composition of the fungal communities were determined using high-throughput sequencing of the internal transcribed spacer. Six grasslands were selected to represent different successional age classes: 0 (cropland), 5, 10, 15, 20, and 30 yr. Short-term decreases (initial 5 yr) in the amounts of soil organic carbon, total nitrogen, available phosphorus, and fungal biomass and in fungal diversity had returned to original levels (i.e., cropland) within 15 yr and were much higher after continued succession. Abandoning cropland for succession caused the soil erodibility (K) decrease and the aboveground coverage, soil nutrient levels, content of larger (>5 mm) water-stable aggregate, mean aggregate weight diameter, and diversity of the fungal communities improvement including arbuscular mycorrhizas (AMF), ectomycorrhizas (EMF), and saprotrophs. The fungal communities were dominated by Ascomycota, Zygomycota, Basidiomycota, and Glomeromycota during the succession. The successional patterns of the plant and fungal communities were similar, although distinct fungal communities were not observed in the two initial stages, suggesting that fungal succession may develop more slowly than plant succession. Plant root biomass, EMF, and soil organic carbon content accounted for most of the variation of soil erodibility (28.6%, 19.5%, and 11.8%, respectively), indicating their importance in shaping soil structure to prevent erosion. Our results demonstrated that abandoning cropland for natural succession could decrease soil erodibility and increase fungal diversity. EMF plays an important role in soil stability against erosion in the Loess Plateau. Abandoning cropland for natural succession should be recommended for alleviating soil erosion and improving the degraded soils in this area.