The Yellow River is one of the major rivers with severe runoff declines in China, but there are significant differences in runoff changes in the upper and lower reaches of the basin and among different tributaries. However, the characteristic of runoff change and its spatial heterogeneity are not well understood in the whole basin. In this paper, 48 hydrological stations located in the mainstream and major tributaries were selected, and the meteorological and runoff data from 1956 to 2017 were collected. The multi-year and intra-year changes in runoff were analyzed, and then the attribution of climate change and human activity to runoff change was quantified by the climate elasticity coefficients. The results showed that: (1) in the past 60 years, the runoff of the Yellow River showed a serious decrease trend of −8.25 mm/10a. Moreover, most tributaries decreased significantly in runoff with a rate of −1.42 mm/10a to −28.99 mm/10a; (2) for the whole basin, the contribution of climate change and human activity to runoff changes was 13% and 87%, respectively. Moreover, the contribution of the two factors varied considerably in different tributaries. Finally, focusing on different runoff regime and socioeconomic characteristics, this study provided corresponding water resources adaptive management suggestions.