The basin along the Yangtze River in Anhui Province is an important ecological protective screen in the Yangtze River Delta Economic Belt, with a large number of wetlands, lakes, and nature reserves in the basin. The effect of the rapid development of regional urbanization on the ecological environment quality has become an important threat source that restricts ecosystem function and biodiversity protection in the basin. Therefore, this study used InVEST model to analyze the spatial and temporal evolution of habitat quality based on remote sensing image data from 1990, 2000, 2010, and 2020 in the basin along the Yangtze River in Anhui Province and revealed the spatial evolution trend of habitat quality degradation by using hot and cold spot analysis methods. The geographical detector model was used to discuss the main driving factors of habitat quality change. The study results showed that a trend of increase and decrease of construction land and paddy land in the basin from 1990 to 2020 was the opposite, that is, the area of construction land increased, and the area of paddy land decreased. Especially, the area of construction land increased from 390.18 km2 in 1990 to 1616.34 km2 in 2020. The area of increase and decrease of other land types remained around 1% to 2%. During the period from 2000 to 2020, Construction land was mainly transferred in from paddy land, accounting for over 60% of the area transferred in, which indicated the continuous increase of human activity intensity in the study area. From 1990 to 2020, the areas with a significant decline in habitat quality in the basin were mainly concentrated along the Yangtze River and in the northern part of the Chaohu Lake. The area proportion with the lowest grade of habitat quality showed a trend of increasing year by year, that is, the area proportion increased from 4.85% in 1990 to 8.47% in 2020. The hot spots of habitat quality degradation were concentrated in Hefei and its surrounding areas, while the cold spots of the degradation were mainly concentrated in the southern and western mountainous areas. Land use type was the main driving factor affecting habitat quality, and the interaction between land use and per capita GDP was the main driving factor for changes in habitat quality. The study results had important theoretical and practical value for the ecological environment protection and harmonious development of the relationship between humans and nature in the basin along the Yangtze River in Anhui Province.