The rural human settlement is an important carrier for rural residents’ production and living life, and its quality level is an important basis for judging the current living conditions of rural residents. This study aims to explore the spatial-temporal characteristics of the rural human settlement in mountainous counties of Zhejiang Province, as well the influencing factors, based on a constructed evaluation index system for a quality improvement in these areas. This study takes the rural human settlement of the mountainous counties in Zhejiang Province as the research object, constructs an evaluation index system of rural human settlement quality of these counties based on the current situation and development policies of Zhejiang Province, and evaluates the rural human settlement quality from 2016 to 2020. Further, spatial autocorrelation analysis, trend surface analysis, and other analysis methods are used to evaluate the results. The evaluation results were analyzed using spatial autocorrelation analysis and trend surface analysis, and the performance of each influencing factor and interaction force was detected using geographic probes. The study found that: (1) Overall, the quality of the rural human settlement in the mountainous counties of Zhejiang Province has been steadily improving over the past five years, and there is no obvious spatial correlation among counties, though the spatial development characteristics and development trend of the north is better than the south, and the east is better than the central areas. (2) Each subsystem also shows a steady increase and presents different spatial distribution characteristics. (3) The quality of the rural human settlement is significantly associated with policy orientation, the level of economic development, and landscape development characteristics. (4) The factors that currently have the greatest effect on the 26 counties include the level of agricultural modernization and economic development level, among which five groups of factors have an interactive effect. In sum, this study judged the development trend of rural human settlement quality and the main influencing factors in the mountainous counties of Zhejiang Province this year and provides a scientific basis for their improvement. It also provides research ideas for the evaluation and optimization of rural habitat quality in other mountainous counties in China.