Plant water‐use efficiency (WUE, the carbon gained through photosynthesis per unit of water lost through transpiration) is a tracer of the plant physiological controls on the exchange of water and carbon dioxide between terrestrial ecosystems and the atmosphere. At the leaf level, rising CO2 concentrations tend to increase carbon uptake (in the absence of other limitations) and to reduce stomatal conductance, both effects leading to an increase in leaf WUE. At the ecosystem level, indirect effects (e.g. increased leaf area index, soil water savings) may amplify or dampen the direct effect of CO2. Thus, the extent to which changes in leaf WUE translate to changes at the ecosystem scale remains unclear. The differences in the magnitude of increase in leaf versus ecosystem WUE as reported by several studies are much larger than would be expected with current understanding of tree physiology and scaling, indicating unresolved issues. Moreover, current vegetation models produce inconsistent and often unrealistic magnitudes and patterns of variability in leaf and ecosystem WUE, calling for a better assessment of the underlying approaches. Here, we review the causes of variations in observed and modelled historical trends in WUE over the continuum of scales from leaf to ecosystem, including methodological issues, with the aim of elucidating the reasons for discrepancies observed within and across spatial scales. We emphasize that even though physiological responses to changing environmental drivers should be interpreted differently depending on the observational scale, there are large uncertainties in each data set which are often underestimated. Assumptions made by the vegetation models about the main processes influencing WUE strongly impact the modelled historical trends. We provide recommendations for improving long‐term observation‐based estimates of WUE that will better inform the representation of WUE in vegetation models.