Livestock grazing is a widespread practice in human activities worldwide. However, the effects of livestock grazing management on vegetation carbon storage have not been thoroughly evaluated. In this study, we used the system dynamic approach to simulate the effects of different livestock grazing management strategies on carbon stock in sparse elm woodlands. The livestock grazing management strategies included rotational grazing every 5 years (RG5), prohibited grazing (PG), seasonal prohibited grazing (SPG), and continuous grazing (CG). We evaluated the carbon sequestration rate in vegetation using logistical models. The results showed that the carbon stock of elm trees in sparse woodlands was 5–15 M g ha−1. The values of the carbon sequestration rate were 0.15, 0.13, 0.13, and 0.09 Mg C ha−1 year−1 in RG5, PG, CG, and SPG management, respectively. This indicates that rotational grazing management might be the optimal choice for improving vegetation carbon accumulation in sparse woodlands. This study contributes to decision-making on how to choose livestock grazing management to maintain higher carbon storage.