The construction of the ecological environment in rural settlements is a very important part of the development of human settlements. Key to this construction is the coordinated and sustainable development of plant environment, geographical environment, and human activities. Therefore, it is necessary to study the characteristics of plant diversity and their driving mechanisms. The study area, ‘Shanchuan Town’, is located in the hilly area of Southeast China, Anji County, Zhejiang Province. As bamboo forests cover 62% of the total forests, it has the reputation of being the “China town of bamboo”. To interpret rural plant community features, we extracted topography data by the field survey, satellite image, and Digital Elevation Model (DEM) based on ArcGIS platform, then used Spearman’s correlation and a redundancy analysis to examine the relationships between explanatory variables (e.g., plot elevation, slope aspect and architectural height, etc.) and the distribution of plant community diversity. In this study, 227 plots were selected from 14 settlements to investigate totally 105, 88, 206, and 17 species of trees, shrubs, herbs, and vines, respectively, belonging to 147 families, 324 genera, and 416 species. Among them, there were 19 bamboo species, and 47 species of alien plants, which accounted for 11.3% of the total. The dominant trees and shrubs were mainly native species with high edible or ornamental value. The dominant bamboo species were common species for bamboo shoots in Zhejiang Province, while most of the dominant herbaceous species were alien species. Among the explanatory variables, the impact of plot elevation (PE) on plant community was significantly higher than those of other explanatory factors. The correlation analysis showed that the richness and diversity indices of different plant life layers were significantly related to PE. Among the architectural factors, the architectural orientation (AO) and layout type (AT) effect on shrubs and other herbaceous species was stronger than those on trees and bamboo. There was a high plant community richness in the study area, and both topography and architectural factors had a significant effect on plant community. Therefore, construction of the plant landscape should conform to the topography and regulate the residential construction activities properly, so as to get rid of urban planning thinking and the excessive pursuit of plant diversity, realizing the sustainable development of ecological environment in settlements.