Many arid and semi‐arid rangelands exhibit distinct spatial patterning of vegetated and bare soil‐dominated patches. The latter potentially represent a grazing‐induced, degraded ecosystem state, but could also arise via mechanisms related to feedbacks between vegetation cover and soil moisture availability that are unrelated to grazing. The degree to which grazing contributes to the formation or maintenance of degraded patches has been widely discussed and modeled, but empirical studies of the role of grazing in their formation, persistence, and reversibility are limited.We report on a long‐term (17 years) grazing removal experiment in a semi‐arid savanna where vegetated patches composed of perennial grasses were interspersed within large (>10 m2) patches of bare soil.Short‐term (3 years) grazing removal did not allow bare patches to become revegetated, whereas following long‐term (17 years) grazing removal, bare soil patches were revegetated by a combination of stoloniferous grasses and tufted bunchgrasses. In the presence of grazers, stoloniferous grasses partially recolonized bare patches, but this did not lead to full recovery or to the establishment of tufted bunchgrasses.These results show that grazers alter both the balance between bare and vegetated patches, as well as the types of grasses dominating both patch types in this semiarid savanna.Synthesis: Large herbivores fundamentally shaped the composition and spatial pattern of the herbaceous layer by maintaining a two‐phase herbaceous mosaic. However, bare patches within this mosaic can recover given herbivore removal over sufficiently long time scales, and hence do not represent a permanently degraded ecosystem state.