The Abuse of Popular Performance Metrics in Hydrologic Modeling

Clark, Martyn P.; Vogel, Richard M.; Lamontagne, Jonathan R.; Mizukami, Naoki; Knoben, Wouter; Tang, Guoqiang; Gharari, Shervan; Freer, Jim; Whitfield, Paul H.; Shook, Kevin; Papalexiou, S.

doi:10.1029/2020wr029001

Cited by 131 publications

(79 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In contrast, the modeling approach described here calibrates to specific aspects of the flow regime, avoiding such tradeoffs, and likely increasing model predictive accuracy of functional flow metrics. Hydrologic models are typically evaluated using a limited set of "goodness of fit" (GOF) criteria, such as r-squared and NSE, to compare predictions with paired observations (Clark et al, 2021). The performance assessment approach used in this study used a broader suite of criteria, including both GOF and measures that evaluate the degree to which the distributions of predictions align with observations.…”

Section: Discussionmentioning

confidence: 99%

“…To assess model performance, we compared predicted FFM values with observations at sites excluded from model training. We restricted the assessment to sites with 20 or more observations (i.e., 20 years of record) and calculated several model performance criteria to limit the risk of flawed interpretation resulting from the use of a single performance metric (Clark et al, 2021). We calculated performance criteria that provided measures of both the dispersion and central tendency of model predictions in comparison to observed values.…”

Section: Functional Flow Componentmentioning

confidence: 99%

See 1 more Smart Citation

Modeling Functional Flows in California’s Rivers

Grantham

Carlisle

Howard

et al. 2022

Front. Environ. Sci.

View full text Add to dashboard Cite

Environmental flows are critical to the recovery and conservation of freshwater ecosystems worldwide. However, estimating the flows needed to sustain ecosystem health across large, diverse landscapes is challenging. To advance protections of environmental flows for streams in California, United States, we developed a statewide modeling approach focused on functional components of the natural flow regime. Functional flow components in California streams—fall pulse flows, wet season peak flows and base flows, spring recession flows, and dry season baseflows—support essential physical and ecological processes in riverine ecosystems. These functional flow components can be represented by functional flow metrics (FFMs) and quantified by their magnitude, timing, frequency, duration, and rate-of-change from daily streamflow records. After calculating FFMs at reference-quality streamflow gages in California, we used machine-learning methods to estimate their natural range of values for all stream reaches in the state based on physical watershed characteristics, and climatic factors. We found that the models performed well in predicting FFMs in streams across a diversity of landscape and climate contexts, according to a suite of model performance criteria. Using the predicted FFM values, we established initial estimates of ecological flows that are expected to support critical ecosystem functions and be broadly protective of ecosystem health. Modeling functional flows at large regional scales offers a pathway for increasing the pace and scale of environmental flow protections in California and beyond.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Functional Flow Componentmentioning

confidence: 99%

Modeling Functional Flows in California’s Rivers

Grantham

Carlisle

Howard

et al. 2022

Front. Environ. Sci.

View full text Add to dashboard Cite

show abstract

“…For both case studies we calibrated the HyMod model by minimizing the Nash‐Sutcliffe efficiency. It is well known that performance metrics are affected by significant sampling uncertainty (Barber et al., 2020; Clark et al., 2021). Lamontagne et al.…”

Section: Case Studiesmentioning

confidence: 99%

“…For both case studies we calibrated the HyMod model by minimizing the Nash-Sutcliffe efficiency. It is well known that performance metrics are affected by significant sampling uncertainty (Barber et al, 2020;Clark et al, 2021). Lamontagne et al (2020) have shown that estimation robustness may be improved by performing a preliminary logarithmic transformation of observed and simulated river flow data.…”

Section: Case Studiesmentioning

confidence: 99%

Bluecat: A Local Uncertainty Estimator for Deterministic Simulations and Predictions

Koutsoyiannis

Montanari

2022

Water Resources Research

View full text Add to dashboard Cite

We present a new method for simulating and predicting hydrologic variables with uncertainty assessment and provide example applications to river flows. The method is identified with the acronym “Bluecat” and is based on the use of a deterministic model which is subsequently converted to a stochastic formulation. The latter provides an adjustment on statistical basis of the deterministic prediction along with its confidence limits. The distinguishing features of the proposed approach are the ability to infer the probability distribution of the prediction without requiring strong hypotheses on the statistical characterization of the prediction error (e.g., normality, homoscedasticity), and its transparent and intuitive use of the observations. Bluecat makes use of a rigorous theory to estimate the probability distribution of the predictand conditioned by the deterministic model output, by inferring the conditional statistics of observations. Therefore Bluecat bridges the gaps between deterministic (possibly physically based, or deep learning‐based) and stochastic models, as well as between rigorous theory and transparent use of data with an innovative and user oriented approach. We present two examples of application to the case studies of the Arno river at Subbiano and Sieve river at Fornacina. The results confirm the distinguishing features of the method along with its technical soundness. We provide an open software working in the R environment, along with help facilities and detailed instructions to reproduce the case studies presented here.

show abstract

“…Thus, GR4J NSE and HBV N SE have higher median NSE efficiencies (0.71 and 0.64, respectively) than GR4J KGE and HBV KGE (0.66 and 0.63, respectively) on the evaluation (hindcast) period. Obtained results raise a question of the best metric that could serve the needs of all interested parties: professional community, government agencies, and general public [49]. Currently, NSE and KGE metrics are popular only within the hydrological community.…”

Section: Consistency Between Calibration and Evaluation Periodsmentioning

confidence: 99%

OpenForecast: An Assessment of the Operational Run in 2020–2021

Ayzel

Abramov

2022

Geosciences

View full text Add to dashboard Cite

OpenForecast is the first openly available national-scale operational runoff forecasting system in Russia. Launched in March 2020, it routinely provides 7-day ahead predictions for 834 gauges across the country. Here, we provide an assessment of the OpenForecast performance on the long-term evaluation period from 14 March 2020 to 31 October 2021 (597 days) for 252 gauges for which operational data are available and quality-controlled. Results show that OpenForecast is a robust system based on reliable data and solid computational routines that secures efficient runoff forecasts for a diverse set of gauges.

show abstract

The Abuse of Popular Performance Metrics in Hydrologic Modeling

Cited by 131 publications

References 78 publications

Modeling Functional Flows in California’s Rivers

Modeling Functional Flows in California’s Rivers

Bluecat: A Local Uncertainty Estimator for Deterministic Simulations and Predictions

OpenForecast: An Assessment of the Operational Run in 2020–2021

Contact Info

Product

Resources

About