As the most populous country in the world, China has a great shortage pressure of water resources. With the acceleration of urbanization, China’s water usage in different sectors will change significantly in next few years. In order to investigate the main reasons behind water usage change in China, the Logarithmic Mean Divisia Index (LMDI) model was adopted in this paper from 2000 to 2020 with provincial data. Three effects, including that of technology, industrial structure, and regional scale, were analyzed. In addition, the decoupling effect between water usage and economic growth was also considered. The results show that: (1) from 2000 to 2020, the technological effect, industrial structure effect, and regional scale effect are −376.54, −89.85 and 20.66, respectively; (2) the technical effect and industrial structure effect have the greatest impact on primary industry, followed by secondary industry; (3) the technical effect is greater than the industrial structure effect in most provinces; and (4) the decoupling state gradually changes from weak decoupling to strong decoupling. In the future, the key policy recommendations for water saving are the following: (1) technological innovation has the most efficient effect on the reduction of water usage in China, and (2) the optimization of industrial structure can be helpful in water-saving in the future.