Evaluating, interpreting, and communicating performance of hydrologic/water quality models considering intended use: A review and recommendations

Harmel, R. Daren; Smith, Patricia K.; Migliaccio, Kati W.; Chaubey, Indrajeet; Douglas-Mankin, Kyle R.; Benham, Brian Leslie; Shukla, Sanjay Kumar; Muñoz‐Carpena, Rafael; Robson, Barbara

doi:10.1016/j.envsoft.2014.02.013

Cited by 132 publications

(98 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These include multicomponent mapping (Pappenberger and Beven, 2004), self-organizing maps (Reusser et al, 2009), wavelets (Liu et al, 2011), the hydrograph matching algorithm (Ewen, 2011), and the "Peak-Box" approach for the interpretation and verification of operational ensemble peakflow forecasts (Zappa et al, 2013). Despite this considerable progress, many practical and scientific applications (Haag et al, 2005;Gassmann et al, 2013;Seibert et al, 2014;Wrede et al, 2014;Kelleher et al, 2015;Zhang et al, 2016) still rely on simple mean squared error (MSE) type distance metrics such as the long-established Nash-Sutcliffe efficiency (NASH) or the root mean squared error (RMSE) even though their shortcomings are well known (Seibert, 2001;Schaefli and Gupta, 2007;Gupta et al, 2009). A less recognized issue of MSE-type criteria is that these compare points with identical abscissa, i.e.…”

Section: Single and Multiple Criteria For Hydrograph Evaluationmentioning

confidence: 99%

“…For this reason different attempts have been undertaken to compare expert judgement and automated criteria (Crochemore et al, 2014) and to establish model evaluation guidelines (e.g. Moriasi et al, 2007;Biondi et al, 2012;Harmel et al, 2014). Key points of related guidelines typically include the statement that the choice of the metric should depend (i) on the modelling purpose, (ii) on the modelling mode (calibration, validation, simulation, or forecast), and (iii) on the model resolution (time stepping, spatial resolution).…”

Section: Single and Multiple Criteria For Hydrograph Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

Disentangling timing and amplitude errors in streamflow simulations

Seibert

Ehret

Zehe

2016

Hydrol. Earth Syst. Sci.

View full text Add to dashboard Cite

Abstract. This article introduces an improvement in the Series Distance (SD) approach for the improved discrimination and visualization of timing and magnitude uncertainties in streamflow simulations. SD emulates visual hydrograph comparison by distinguishing periods of low flow and periods of rise and recession in hydrological events. Within these periods, it determines the distance of two hydrographs not between points of equal time but between points that are hydrologically similar. The improvement comprises an automated procedure to emulate visual pattern matching, i.e. the determination of an optimal level of generalization when comparing two hydrographs, a scaled error model which is better applicable across large discharge ranges than its nonscaled counterpart, and "error dressing", a concept to construct uncertainty ranges around deterministic simulations or forecasts. Error dressing includes an approach to sample empirical error distributions by increasing variance contribution, which can be extended from standard one-dimensional distributions to the two-dimensional distributions of combined time and magnitude errors provided by SD.In a case study we apply both the SD concept and a benchmark model (BM) based on standard magnitude errors to a 6-year time series of observations and simulations from a small alpine catchment. Time-magnitude error characteristics for low flow and rising and falling limbs of events were substantially different. Their separate treatment within SD therefore preserves useful information which can be used for differentiated model diagnostics, and which is not contained in standard criteria like the Nash-Sutcliffe efficiency. Construction of uncertainty ranges based on the magnitude of errors of the BM approach and the combined time and magnitude errors of the SD approach revealed that the BM-derived ranges were visually narrower and statistically superior to the SD ranges. This suggests that the combined use of time and magnitude errors to construct uncertainty envelopes implies a trade-off between the added value of explicitly considering timing errors and the associated, inevitable time-spreading effect which inflates the related uncertainty ranges. Which effect dominates depends on the characteristics of timing errors in the hydrographs at hand. Our findings confirm that Series Distance is an elaborated concept for the comparison of simulated and observed streamflow time series which can be used for detailed hydrological analysis and model diagnostics and to inform us about uncertainties related to hydrological predictions.

show abstract

Section: Single and Multiple Criteria For Hydrograph Evaluationmentioning

confidence: 99%

Section: Single and Multiple Criteria For Hydrograph Evaluationmentioning

confidence: 99%

Disentangling timing and amplitude errors in streamflow simulations

Seibert

Ehret

Zehe

2016

Hydrol. Earth Syst. Sci.

View full text Add to dashboard Cite

show abstract

“…• Most studies did not report any estimate of measurement uncertainty (i.e., uncertainty due to an instrument or method), even though researchers are increasingly calling for and explaining the benefits of publishing such values corresponding to measured data (Beven, 2006;Harmel et al, 2006aHarmel et al, , 2009Harmel et al, , 2014). …”

Section: Challenges and Issues In Systematic Reviews Syntheses And mentioning

confidence: 99%

Meta‐Analysis Constrained by Data: Recommendations to Improve Relevance of Nutrient Management Research

et al. 2017

View full text Add to dashboard Cite

Five research teams identifi ed parallel obstacles when concurrently attempting to conduct meta-analyses on the air and water quality impacts of on-farm 4R nutrient management practices. Across projects, system complexity and the lack of relevant data from cultivated and grassland agriculture fi eld trials impeded the application of standard meta-analytical procedures. Because challenges were comparable across projects, the 4R Research Fund technical leadership tasked the researchers with recommending improvements in fi eld research design, data collection, and reporting to enhance future agri-environmental data syntheses and meta-analyses. Here we outline statistical and analytical issues unique to meta-analysis and data synthesis in agriculture, discuss critical data and reporting gaps in the existing literature, and provide specifi c recommendations for researchers, funders, and journals. Key obstacles developed when fi eld studies did not include complete descriptive or response data (per treatment and experiment year), measurement uncertainty, estimation error in treatment eff ects, or simultaneously measured nutrient losses and crop yield. Others did not report crop nutrient uptake or their apparent recovery effi ciencies. To alleviate such challenges for subsequent research, we make the following recommendations: (i) use common meta-data protocols for consistent units and terminology; (ii) clearly defi ne treatments and controls; (iii) provide complete, tabular, full-factorial response data for each year and location; (iv) collect and report a minimum set of auxiliary data; and (v) establish requirements for data curation and repositories in funding and publication cycles. Implementing these in future nutrient management research will facilitate more robust meta-analyses and other data synthesis eff orts.A.J. Eagle, Environmental Defense Fund,

show abstract

“…We recommend comparing observed and simulated time series data in two ways (e.g., van Werkhoven et al, 2009;Hrachowitz et al, 2014): as statistical metrics, which measure model performance with respect to the entire time series, e.g., root mean squared error (RMSE), Nash-Sutcliffe efficiency coefficient (NSE; Nash and Sutcliffe, 1970), and dynamic metrics, which measure model performance with respect to different periods or types of hydrologic behavior, e.g., the baseflow index, the slope of the flow duration curve (Wagener et al, 2001;Gupta et al, 2008;Pfannerstill et al, 2014;Shafii and Tolson, 2015). Performance across statistical metrics is typically judged with respect to a threshold value, e.g., NSE greater than 0.8, or some threshold percentage, e.g., top 10 % of RMSE values (e.g., Moriasi et al, 2007;Harmel et al, 2014). Dynamic metrics may expand assessment of hydrologic behavior, as existing work has shown that there is information contained not only in different types of data but also in different periods for an observational time series (Wagener et al, 2001;Gupta et al, 2008;Pfannerstill et al, 2014;Shafii and Tolson, 2015).…”

Section: Frameworkmentioning

confidence: 99%

Characterizing and reducing equifinality by constraining a distributed catchment model with regional signatures, local observations, and process understanding

Kelleher

McGlynn

Wagener

2017

Hydrol. Earth Syst. Sci.

View full text Add to dashboard Cite

Abstract. Distributed catchment models are widely used tools for predicting hydrologic behavior. While distributed models require many parameters to describe a system, they are expected to simulate behavior that is more consistent with observed processes. However, obtaining a single set of acceptable parameters can be problematic, as parameter equifinality often results in several "behavioral" sets that fit observations (typically streamflow). In this study, we investigate the extent to which equifinality impacts a typical distributed modeling application. We outline a hierarchical approach to reduce the number of behavioral sets based on regional, observation-driven, and expert-knowledge-based constraints. For our application, we explore how each of these constraint classes reduced the number of "behavioral" parameter sets and altered distributions of spatiotemporal simulations, simulating a well-studied headwater catchment, Stringer Creek, Montana, using the distributed hydrology-soil-vegetation model (DHSVM). As a demonstrative exercise, we investigated model performance across 10 000 parameter sets. Constraints on regional signatures, the hydrograph, and two internal measurements of snow water equivalent time series reduced the number of behavioral parameter sets but still left a small number with similar goodness of fit. This subset was ultimately further reduced by incorporating pattern expectations of groundwater table depth across the catchment. Our results suggest that utilizing a hierarchical approach based on regional datasets, observations, and expert knowledge to identify behavioral parameter sets can reduce equifinality and bolster more careful application and simulation of spatiotemporal processes via distributed modeling at the catchment scale.

show abstract

Evaluating, interpreting, and communicating performance of hydrologic/water quality models considering intended use: A review and recommendations

Cited by 132 publications

References 46 publications

Disentangling timing and amplitude errors in streamflow simulations

Disentangling timing and amplitude errors in streamflow simulations

Meta‐Analysis Constrained by Data: Recommendations to Improve Relevance of Nutrient Management Research

Characterizing and reducing equifinality by constraining a distributed catchment model with regional signatures, local observations, and process understanding

Contact Info

Product

Resources

About