The water resource situation in China is severe, and conflicts between the supply and demand of water resources are prominent. Competition for water from key sectors, such as agriculture, industry, and domestic use, is widespread. The Yellow River, as one of the longest rivers in the world, is an important economic belt and an ecological barrier in China. This study considered the nine provinces along the Yellow River as the study area and the three major water-use sectors: agriculture, industry, and domestic as the research objects. The drivers of water consumption in each sector in the nine provinces along the Yellow River were analyzed using the Logarithmic Mean Divisia Index method. Based on this, a decoupling model was used to explore the relationship between water use in each sector and the corresponding level of economic development. It was found that water use intensity and economic development level were the largest negative and positive influencing factors on water use in each sector, respectively, and the opposite effects of the two may cause the Jevons paradox in water use. The overall agricultural water-saving level in the basin is high and has a large water-saving potential. The negative driving effect of the industrial structure was more significant in provinces with higher development levels. The positive driving effect of residents' consumption levels on domestic water use in rural areas was more obvious than in urban areas. The degree of decoupling between per capita and domestic water consumption in urban areas was the worst in the decoupling of water use in the agricultural, industrial, and domestic sectors and their corresponding levels of economic development. Therefore, focusing on areas with weak agricultural water conservation, promoting industrial structure upgrading, strengthening water conservation education in rural areas, and guiding the water-saving consumption habits of residents can promote the sustainable development of water resources in the provinces along the Yellow River. The research results provide insights into water conservation management in the Yellow River Basin.