Vegetation growth and its response to climatic factors have become one of the most pressing issues in ecological research. However, no consensus has yet been reached on how to resolve this problem in arid areas with a high-elevation gradient and complex underlying surface. Here, NOAA CDR AVHRR NDVI V5 for 1981–2018 and China’s regional surface meteorological faction-driven datasets were used. General linear regression, the Mann-Kendall test and sliding t-test, Pearson correlations, and the Akaike information criterion (AIC), on a grid-scale, were applied to analyze the annual normalized difference vegetation index (NDVI) and its relationship with temperature and precipitation in the Altay region. Results revealed that the temporal trend of NDVI for most grid cells was non-significant. However, mountains, coniferous forests, grasslands, and meadows in the high-elevation zone displayed a slow increasing trend in NDVI. Further, NDVI was positively correlated with the mean annual temperature and total annual precipitation, the latter playing a more significant role. Yet, for desert and shrub vegetation and coniferous forest, their NDVI had insignificant negative correlations with the mean annual temperature. Hence, both the trends and drivers of NDVI of high elevation are highly complex. This study’s findings provide a reference for research on vegetation responses to climate change in arid areas having a high-elevation gradients and complex underlying surfaces.