The growing severity of farmland abandonment in rural areas has sparked concerns about the lack of people willing to cultivate the land. This study attempts to find the endogenous factors of farmland land abandonment among Chinese farmers and to curb land degradation to stabilize food security. Based on the China Labor-force Dynamics Survey (2016–2018), this paper explores the effect of famine experience on farmland abandonment using logit and DID models. We find that famine experience does increase the probability and extent of farmland abandonment, and undergo a series of robustness tests. We also find that the famine stigma effect increases the likelihood and extent of farmland abandonment by increasing risk aversion and reducing social trust and investment in agriculture. The growth of urbanization in China worsens the effects of famine, whereas local communal production services remove the effects of famine. The overall findings of this study provide useful insights for the government to protect land quality and food security and to carefully consider the role played by communities.