Studying the variation in vegetation types within the source region of the Yellow River (SRYR) is of great significance for understanding the response of vegetation to climate change and human activities on the Qinghai-Tibet Plateau (QTP) permafrost. In order to understand the characteristics of the variation in vegetation associations in the SRYR under the influence of climate and human activities, two hyperspectral remote sensing images from HJ-1A in 2013 and OHS-3C in 2020 were used to extract the vegetation types located in the area south of Ngoring Lake, covering 437.11 km2 in Maduo County, from the perspective of vegetation associations. Here, the hybrid spectral CNN (HybridSN) model, which is dependent on both spatial and spectral information, was used for vegetation association classifications. On this basis, the variations in vegetation associations from 2013 to 2020 were studied using the transition matrix, and the variation in noxious weeds across different altitude and slope gradients was analyzed. As an example, Thermopsis lanceolata’s spatial distribution pattern and diffusion mechanism were analyzed. The results showed that (1) in addition to noxious weeds, herbage such as Poa poophagorum, Stipa purpurea, Kobresia humilis, and Carex moorcroftii increased, indicating that the overall ecological environment tended to improve, which may be attributed mainly to the development of a warm and humid climate. (2) Most of the noxious weeds were located at low altitudes with an area increase in the 4250–4400 m altitude range and a decrease in the 4400–4500 m altitude range. More attention should be given to the fact that the noxious weeds area increased from 2.88 km2 to 9.02 km2 between 2013 and 2020, which was much faster than that of herbage and may threaten local livestock development. (3) The Thermopsis lanceolate association characterized by an aggregated distribution tended to spread along roads, herdsmen sites, and degraded swamps, which were mainly affected by human activities and swamp degradation.