Agricultural water use accounts for the largest proportion of water withdrawal, so improving agricultural water use efficiency is an important way to alleviate water shortage. However, the expected water saving by the improved agricultural water use efficiency may be offset by the rebound effect, which means the goal of water saving by improving agricultural water use efficiency is not achieved. Based on the definition of the rebound effect of agricultural water use, this paper first uses a fixed model to measure the causal effect of agricultural water use efficiency on agricultural water use to analyze the agricultural water rebound effect, then analyses the heterogeneity and mechanism of the effect of agricultural water use efficiency on agricultural water use with the panel data from 30 provinces or cities in China from 2000 to 2017. The results show that, firstly, the agricultural water use efficiency has a significant negative effect on agricultural water use, but the average agricultural water rebound effect is 88.81%. Secondly, the effect of agricultural water use efficiency on agricultural water use is heterogeneous, in which the improvement of agricultural water use efficiency in humid or major grain-producing areas will have a lower agricultural water rebound effect. Finally, agricultural water use efficiency can affect agricultural water use through planting area and planting structure. An increase in agricultural water use efficiency will expand the planting area to increase water use. However, this will change the planting structure to decrease water use. The implication for agricultural water management is that the irrigation agricultural scale has to be controlled under the condition of available water resource, while improving agricultural water use efficiency.