Ecological environment assessment would be helpful for a rapid and systematic understanding of ecological status and would contribute to formulate appropriate strategies for the sustainability of heritage sites. A procedure based on spatial principle component analysis was employed to measure the ecological status in Bayinbuluke; exploratory spatial data analysis and geo-detector model were introduced to assess the spatio-temporal distribution characteristics and detect the driving factors of the ecological environment. Five results are presented: (1) During 2007–2018, the average values of moisture, greenness, and heat increased by 51.72%, 23.10%, and 4.99% respectively, and the average values of dryness decreased by 56.70%. However, the fluctuation of each indicator increased. (2) The ecological environment of Bayinbuluke was improved from 2007 to 2018, and presented a distribution pattern that the heritage site was better than the buffer zone, and the southeast area was better than the northwest area. (3) The ecological environment presented a significant spatial clustering characteristic, and four types of spatial associations were proposed for assessing spatial dependence among the samples. (4) Elevation, protection partition, temperature, river, road, tourism, precipitation, community resident, and slope were statistically significant with respect to the changes in ecological status, and the interaction of any two factors was higher than the effect of one factor alone. (5) The remote-sensing ecological index (RSEI) could reflect the vegetation growth to a certain extent, but has limited ability to respond to species structure. Overall, the framework presented in this paper realized a visual and measurable approach for a detailed monitoring of the ecological environment and provided valuable information for the protection and management of heritage sites.