Perennial cropping has been an alternative land use type due to its widely accepted role in increasing soil carbon sequestration. However, how soil organic carbon (SOC) changes and its underlying mechanisms under different cropping years are still elusive. A chronosequence (0-, 3-, 6-, 20-year) of perennial mugwort cropping was chosen to explore the SOC dynamics and the underlying mechanisms in agricultural soils of Northern China Plain. The results revealed that SOC first increased and then decreased along the 20-year chronosequence. The similar patterns were also found in soil properties (including soil ammonium nitrogen, total nitrogen and phosphorus) and two C-degrading hydrolytic enzyme activities (i.e., α-glucosidase and β-glucosidase). The path analysis demonstrated that soil ammonium nitrogen, total nitrogen, and plant biomass affected SOC primarily through the indirect impacts on soil pH, total phosphorus availability, and C-degrading hydrolytic enzyme activities. In addition, the contributions of soil properties are greater than those of biotic factors (plant biomass) to changes in SOC across the four mugwort cropping years. Nevertheless, the biotic factors may play more important roles in regulating SOC than abiotic factors in the long run. Moreover, SOC reached its maximum and was equaled to that under the conventional rotation when cropping mugwort for 7.44 and 14.88 years, respectively, which has critical implications for sustainable C sequestration of agricultural soils in Northern China Plain. Our observations suggest that short-term but not long-term perennial mugwort cropping is an alternative practice benefiting soil C sequestration and achieving the Carbon Neutrality goal in China.