In Defense of Metrics: Metrics Sufficiently Encode Typical Human Preferences Regarding Hydrological Model Performance

Abstract

Building accurate rainfall-runoff models is an integral part of hydrological science and practice. The variety of modeling goals and applications have led to a large suite of evaluation metrics for these models. Yet, hydrologists still put considerable trust into visual judgment, although it is unclear whether such judgment agrees or disagrees with existing quantitative metrics. In this study, we tasked 622 experts to compare and judge more than 14,000 pairs of hydrographs from 13 different models. Our results show that expert opinion broadly agrees with quantitative metrics and results in a clear preference for a Machine Learning model over traditional hydrological models. The expert opinions are, however, subject to significant amounts of inconsistency. Nevertheless, where experts agree, we can predict their opinion purely from quantitative metrics, which indicates that the metrics sufficiently encode human preferences in a small set of numbers. While there remains room for improvement of quantitative metrics, we suggest that the hydrologic community should reinforce their benchmarking efforts and put more trust in these metrics.

In Defense of Metrics: Metrics Sufficiently Encode Typical Human Preferences Regarding Hydrological Model Performance

“…The collected responses as well as the code to generate all results and tables in this manuscript are publicly available at https://doi.org/10.5281/zenodo.7640347 (Gauch et al, 2023) for others to conduct additional analyses in future work. The hydrographs from the GRIP-GL study are available at https://doi.org/10.20383/103.0598 (see package "A5" for observations and package "A7" for modeled streamflow; Mai et al, 2022a).…”

Benchmarking hydrological models for an uncertain future

“…I have, of course, argued for a long, long time that the idea of an optimum hydrological model should be considered as untenable in favour of a concept of equifinality of models and parameter sets (e.g., Beven, 2006; Beven & Freer, 2001). Others have also suggested that the use of multiple metrics can reflect subjective judgements about the acceptability of different models (e.g., Gauch et al, 2022), though different experts might vary in their rankings (Crochemore et al, 2015).…”

“…Others have also suggested that the use of multiple metrics can reflect subjective judgements about the acceptability of different models (e.g., Gauch et al, 2022), though different experts might vary in their rankings (Crochemore et al, 2015).…”

“…Benchmarking has also been applied to land surface models, including for projects such as PILPS and PLUMBER (Abramowitz, 2012;Best et al, 2015;Haughton et al, 2016;Henderson-Sellers et al, 1996). More recently, model intercomparison and benchmarking projects have included DMIP and IHM-MIP projects for distributed models (e.g., Kollet et al, 2017;Maxwell et al, 2014;Smith et al, 2004Smith et al, , 2012Smith et al, , 2013; the Great Lakes Model Intercomparison project (e.g., Mai et al, 2022); benchmarking of NLDAS land surface models (e.g., Nearing et al, 2016Nearing et al, , 2018; and the testing of model ensembles (Pappenberger et al, 2015). These have taken the form either of testing which model provides the best simulations according to some metric (often using a split record test, e.g., Knoben et al, 2019); or testing against a benchmark model, either a chosen conceptual hydrological model (e.g., Newman et al, 2017;Seibert et al, 2018) or a purely data-based or machine learning model (e.g.…”

Benchmarking Hydrological Models for an Uncertain Future

“…Others have also suggested that the use of multiple metrics can reflect subjective judgments about the acceptability of different models (e.g. Gauch et al, 2022), though different experts might vary in their rankings (Crochemore et al, 2015).…”