Confrontation of climate models with observationally-based reference datasets is widespread and integral to model development. These comparisons yield skill metrics quantifying the mismatch between simulated and reference values and also involve analyst choices, or meta-parameters, in structuring the analysis. Here, we systematically vary five such meta-parameters (reference dataset, spatial resolution, regridding approach, land mask, and time period) in evaluating evapotranspiration (ET) from eight CMIP5 models in a factorial design that yields 68 700 intercomparisons. The results show that while model-data comparisons can provide some feedback on overall model performance, model ranks are ambiguous and inferred model skill and rank are highly sensitive to the choice of meta-parameters for all models. This suggests that model skill and rank are best represented probabilistically rather than as scalar values. For this case study, the choice of reference dataset is found to have a dominant influence on inferred model skill, even larger than the choice of model itself. This is primarily due to large differences between reference datasets, indicating that further work in developing a community-accepted standard ET reference dataset is crucial in order to decrease ambiguity in model skill.

<p><strong>Abstract.</strong> Significant changes in the water cycle are expected under current global environmental change. Robust assessment of these changes at global scales is confounded by shortcomings in the observed record. Modeled assessments yield conflicting results which are linked to differences in model structure and simulation protocol. Here we compare simulated runoff from six terrestrial biosphere models (TBMs), five reanalysis products, and one gridded surface station product with observations from a network of stream gauges in the contiguous United States (CONUS) from 2001 to 2005. We evaluate the consistency of simulated runoff with stream gauge data at the CONUS and water resource region scale, as well as examining similarity across TBMs and reanalysis products at the grid cell scale. Mean runoff across all simulated products and regions varies widely (range: 71–356 mm yr^-1) relative to observed continental-scale runoff (209 mm yr^-1). Across all 12 products only two are within 10% of the observed value and only four exhibit Nash–Sutcliffe efficiency values in excess of 0.8. Region-level mismatch exhibits a weak pattern of overestimation in western and underestimation in eastern regions; although two products are systematically biased across all regions. In contrast, bias in a temporal sense, within region by water year, is highly consistent. Although gridded composite TBM and reanalysis runoff show some regional similarities for 2001–2005 with CONUS means, individual product values are highly variable. To further constrain simulated runoff and to link model-observation mismatch to model structural characteristics would require watershed-level simulation studies coupled with river routing schemes, standardized forcing data, and explicit consideration of water cycle management.</p>