Actively promoting the orderly and voluntary withdrawal of idle rural house bases and effectively activating “sleeping” land assets are considered important measures to accelerate the modernization of agriculture and rural areas, as well as promoting the integrated development of urban and rural areas. However, few studies have focused on the superimposed effects of negative psychological and social environmental factors on the willingness of farmers to withdraw from their homestead. Therefore, we conducted this study in order to clarify the logical ideas of rural home base withdrawal, analyze the factors that influence the decision of home base withdrawal behavior of interest subjects, and provide a scientific decision basis for promoting rural home base withdrawal and accelerating the process of transferring citizens of agricultural population in terms of policies, measures, and paths, so as to design and develop an incentive mechanism for home base withdrawal of citizens of agricultural transfer population. The results of the study show that: (1) the findings indicate that social deprivation, economic deprivation, and emotional deprivation all significantly and negatively affect farmers’ willingness to withdraw from their homesteads; (2) social trust plays a mediating role between relative deprivation and farmer homestead withdrawal behavior; and (3) urban integration plays a moderating role between social and emotional deprivation and the social trust of farmers, but does not have a significant moderating role in the relationship between economic deprivation and social trust. Furthermore, it plays a moderating role between all deprivation factors (i.e., economic, social, and emotional deprivation) and farmer homestead withdrawal behavior. This study aims to provide useful guidance and policy suggestions for optimizing policies related to farmer homestead withdrawal behaviors, and for scientifically designing the homestead withdrawal mechanism and policy system.