Rural settlement consolidation (RSC) has a critical role in facilitating the transformation of human settlement and land use transition in the rural revitalization process. RSC involves a diversity of stakeholder groups with complex and intertwined concerns. It is therefore crucial to identify the key stakeholders and their main concerns to effectively align rural planning and policymaking. However, this line of research remains underdeveloped. This study provides a novel and holistic network perspective for unpacking the complex relationships among different stakeholders. The results indicate: (1) the network of stakeholder concerns is relatively sparse, with 68 concern nodes and 159 concern ties; (2) The village committee, centralized residents, and contractors occupy the core position within the concerns network, while the local government has the majority of strongly connected nodes; (3) The lists of prominent concern nodes and ties are identified by different network indices, including the degree difference, the out-status centrality, closeness centrality, node betweenness centrality, and link betweenness centrality; (4) The main interaction type among stakeholder groups can be classified into five categories: financing, psychological attachment, stakeholder participation, project management, and the improvement in living conditions and infrastructure. This study reveals the relatively weak status of residents, the pivotal role of the village committee, as well as the indispensable part of the contractor and township government, with the aim to provide targeted guidance and decision-making supports for strengthening interactions and cooperation among different stakeholder groups. The findings shed new light on performing the multi-tasks of RSC and facilitating the sustainable management of rural areas.