With the advancement of urbanization, the ecological environmental changes caused by the continuous expansion of cities have become a major concern. Thus, this study is based on a remote sensing image map of Hefei city from 1984 to 2020, which is now in the process of rapid development. We constructed an index system with ecological land, ecosystem service value, ecological environment quality, urban land expansion coefficient, urban heat island rate index, total pollutant emissions, ecological pressure, and ecological deficit, and then evaluated the ecological environment of Hefei city with the help of the entropy weight comprehensive evaluation method. The results show that (1) in the positive evaluation index, the ecological land area was reduced to half of the original area, the ecological service value was reduced by 0.37 times its initial value, and the eco-environmental index was <35 and still decreasing. (2) In the negative evaluation index, the urban heat island ratio index increased by nearly 10 times its initial value, the total emission of various pollutants increased by 6.64 times its initial value, the ecological deficit increased year by year after 1999, and the ecological pressure in 2020 increased to 3 times that in 1999. (3) The overall environmental score of the Hefei urban area has decreased by 48.6 times its initial value. The values of positive evaluation indexes are decreasing, whereas the values of negative evaluation indexes are increasing, and both these items are moving backward, leading to the decrease in the comprehensive ecological environmental score. It can be noticed that the urban expansion of Hefei has had a severe impact on the ecological environment, resulting in the continuous reduction in the ecological land area, a serious urban heat island effect, and continuous weakening of regional ecological service functions. The study results can be used for reference in alleviating the impacts of urban expansion on the environment, promoting the optimization of the urban landscape and the sustainable development of the social economy and environment.