Shoot and root litter are two major sources of soil organic carbon, and their decomposition is a crucial nutrient cycling process in the ecosystem. Altitude and land use could affect litter decomposition by changing the environment in mountain grassland ecosystems. However, few studies have investigated the effects of land use on litter decomposition in different altitudes. We examined how land-use type (mowing vs. grazing) affected shoot and root litter decomposition of a dominant grass (Bromus inermis) in mountain grasslands with two different altitudes in northwest China. Litterbags with 6 g of shoot or root were fixed in the plots to decompose for one year. The mass loss rate of the litter, and the environmental attributes related to decomposition, were measured. Litter decomposed faster in mowing than grazing plots, resulting from the higher plant cover and soil moisture but lower bulk density, which might promote soil microbial activities. Increased altitude promoted litter decomposition, and was positively correlated with soil moisture, soil organic carbon (SOC), and β-xylosidase activity. Our results highlight the diverse influences of land-use type on litter decomposition in different altitudes. The positive effects of mowing on shoot decomposition were stronger in lower than higher altitude compared to grazing due to the stronger responses of the plant (e.g., litter and aboveground biomass) and soil (e.g., soil moisture, soil bulk density, and SOC). Soil nutrients (e.g., SOC and soil total nitrogen) seemed to play essential roles in root decomposition, which was increased in mowing plots at lower altitude and vice versa at higher altitude. Therefore, grazing significantly decreased root mass loss at higher altitude, but slightly increased at lower altitude compared to mowing. Our results indicated that the land use might variously regulate the innate differences of the plant and edaphic conditions along an altitude gradient, exerting complex impacts in litter decomposition and further influencing carbon and nutrient cycling in mountain grasslands.