Grazing leads to the reduction of biomass and plays a critical role in land degradation in arid and semiarid lands. However, the indirect effects of grazing on the ecosystem, e.g., the effect on seed dispersal, have not been well understood. In this study, we built an agent-based model (ABM) to simulate how grazing intensity affects the seed dispersal of elm trees, one of the native vegetation species of temperate woodlands in semiarid lands. The simulated results from the ABM and observed data from the real world were compared to assess the accuracy and validity of the ABM. The results show that elm seed densities in non-grazing, light, moderate, and heavy grazing lands were 74.97 ± 1.44, 57.63 ± 0.89, 37.73 ± 0.95, and 0.97 ± 0.05 seeds m−2, respectively—an apparently decreasing trend. Moreover, as grazing intensity increased, the values of nugget, sill, and partial sill decreased and the value of the ratio of nugget to sill increased. This study indicates that the grazing indirectly leads to the reduction of elm seed density and the increase of spatial heterogeneity of elm seed on the ground in sparse elm woodlands. Moreover, values of geostatistical indices from the ABM were not significantly different from field observation data except for the ratio of nugget to sill. It shows that ABMs can reasonably replicate the spatial pattern of elm seed densities in the field and thus are useful for simulating long-distance seed dispersal in sandy lands. This finding suggests that the indirect effects of grazing should be considered to effectively protect sparse elm woodlands.