The rural settlements in poverty-stricken mountainous areas are the "living fossils" of an economic society with the characteristics of spatial dispersion and are slowly changing. Spatial agglomeration is the development direction of rural settlements. In-depth exploration of the spatial agglomeration characteristics and influencing factors of rural settlements in poverty-stricken mountainous areas is a way to provide a basis for rural settlement restructuring. We selected Pengshui County, a national poverty-stricken county in the southwestern mountainous area of China, as the research area. Spatial buffer and kernel density analysis were used to analyze the agglomeration characteristics of rural settlements and influencing factors. The results show that: (1) The rural settlements are small in scale and the space is evenly dispersed. 55.63% of the rural settlements’ sizes are less than 1000 m2, 84.15% of the rural settlements’ sizes are less than 2500 m2, and 92.81% of the rural settlements are within 200 m. (2) The elevation and slope of topographic factors have a significant agglomeration effect on rural settlements. However, the slope direction has no agglomeration effect. 85.41% of rural settlements (52.75% of rural settlements are gathered between 400 and 800 m above sea level) are gathered at an altitude of 1000 m or less, and 77.59% of rural settlements are gathered with a slope of 6~25°. Additionally, there are few rural settlements with a slope of 0~2°. Moreover, the distribution of residential areas has no agglomeration effect on rural settlements. (3) The cultivated land exerts the most significant effect on rural settlements followed by roads and water sources, while the role of urban land is weak. 99.48% of rural settlements are concentrated in the 100 m area of cultivated land. Therefore, in the poverty-stricken mountainous areas in the southwestern mountainous areas of China, convenient farming is the primary condition for production and living. Rural settlements are highly correlated with cultivated land. Rural settlements are scattered and concentrated with the scattered cultivated land. The rural settlements were leaded by the distribution of cultivated land. Less high-quality cultivated land with less slope were occupied or not by rural residential areas’ people.