Understanding the drivers of forest aboveground biomass (AGB) is essential to further understanding the forest carbon cycle. In the upper Yangtze River region, where ecosystems are incredibly fragile, the driving factors that make AGB changes differ from other regions. This study aims to investigate AGB’s spatial and temporal variation of Pinus densata in Shangri-La and decompose the direct and indirect effects of spatial attribute, climate, stand structure, and agricultural activity on AGB in Shangri-La to evaluate the degree of influence of each factor on AGB change. The continuous sample plots from National Forest Inventory (NFI) and Landsat time series were used to estimate the AGB in 1987, 1992, 1997, 2002, 2007, 2012, and 2017. The structural equation model (SEM) was used to analyze the different effects of the four factors on AGB based on five scales: entire, 1987–2002, 2007–2017, low population density, and high population density. The results are as follows: (1) The AGB of Pinus densata in Shangri-La decreased from 1987 to 2017, with the total amount falling from 9.52 million tons to 7.41 million tons, and the average AGB falling from 55.49 t/ha to 40.10 t/ha. (2) At different scales, stand structure and climate were the drivers that directly affect the AGB change. In contrast, the agricultural activity had a negative direct effect on the AGB change, and spatial attribute had a relatively small indirect effect on the AGB change. (3) Analyzing the SEM results at different scales, the change of the contribution of the agricultural activity indicates that human activity is the main negative driver of AGB change in Shangri-La, especially at the high population density region. In contrast, the change of the contribution of the stand structure and climate indicates that the loss of old trees has an important influence on the AGB change. Forest resources here and other ecologically fragile areas should be gradually restored by adhering to policies, such as strengthening forest protection, improving forest stand quality, and limiting agricultural production activities.