Although afforestation of nitrogen‐fixing shrubs, especially Caragana korshinskii in semiarid areas, is expected to improve ecosystem restoration and carbon sequestration, water shortage induced by these thirsty shrubs may negate potential benefits. Yet there is poor understating of the persistence of soil carbon sequestration in nitrogen‐fixing shrub plantations. Here, we selected three C. korshinskii plantations with different ages (10, 20, and 30 years) on the Loess Plateau to evaluate the persistence of soil carbon sink. We examined dynamics of soil carbon, total nitrogen, and soil water, as well as their relationships along the plantation ages. The results showed that soil organic carbon (SOC) and soil total nitrogen (STN) were tightly coupled. SOC, STN, and soil water stocks (0–100 cm) were significantly decreased by 23.9%, 19.9%, and 29.2% from 10 to 30 years, respectively. The reduction of SOC and STN was likely caused by lowered intercanopy herbaceous productivity due to soil drying and intensified competition from shrubs. We present evidence that the positive carbon sequestration by afforestation of leguminous shrub in short term may be further weakened by water shortage, making our ability to restore degraded land in semiarid regions more challenging than previously thought. Therefore, it is essential to improve management of afforestation land such as thinning for maintaining SOC sequestration persistently in water‐limited areas.