Part-time farming is a widespread phenomenon associated with the long-term global trend of urbanization, especially in China since its reform and opening-up in 1978. The shift of agricultural labor to non-agricultural sectors has significantly impacted cultivated land use activities, yet the connection between part-time farming and cultivated land use sustainability (CLS) remains understudied. Here, we construct an index system for assessing CLS that integrates ecological, economic, and social sustainability. Using survey data from seven Chinese villages across three provinces, we analyze the impact pattern and mechanism of part-time farming on CLS. We find the following: (1) The impact of part-time farming on CLS presents an inverted U-shape, peaking negatively at a 45% inflection point; (2) Spatial heterogeneity exists in the effect of part-time farming on CLS; (3) A household’s non-agricultural workforce size and the gender of the household head significantly moderate the link between part-time farming and CLS; (4) CLS strongly hinges on various factors including the household head’s health, other family members’ education levels, commercial insurance, and agricultural skills training. Our findings provide empirical insights into governing part-time farming for sustainable cultivated land use and, eventually, rural human–land system sustainability.