A Value‐Based Model Selection Approach for Environmental Random Variables

Abstract

Environmental decisions with substantial social and environmental implications are regularly informed by model predictions, incurring inevitable uncertainty. The selection of a set of model predictions to inform a decision is usually based on model performance, measured by goodness‐of‐fit metrics. Yet goodness‐of‐fit metrics have a questionable relationship to a model's value to end users, particularly when validation data are themselves uncertain. For example, decisions based on flow frequency models are not necessarily improved by adopting models with the best overall goodness of fit. We propose an alternative model evaluation approach based on the conditional value of sample information, first defined in 1961, which has found extensive use in sampling design optimization but which has not previously been used for model evaluation. The metric uses observations from a validation set to estimate the expected monetary costs associated with model prediction uncertainties. A model is only considered superior to alternatives if (i) its predictions reduce these costs and (ii) sufficient validation data are available to distinguish its performance from alternative models. By describing prediction uncertainties in monetary terms, the metric facilitates the communication of prediction uncertainty by end users, supporting the inclusion of uncertainty analysis in decision making.

Complementary Vantage Points: Integrating Hydrology and Economics for Sociohydrologic Knowledge Generation

“…Hydroeconomic models integrate the economic notions of scarcity and value by representing water demand as an explicit function of water availability rather than a fixed volumetric requirement (Harou et al, 2009). These insights allowed researchers to assess, for instance, the potential for markets to mitigate the effect of climate variability (Characklis et al, 1999), to evaluate the effectiveness of water use efficiency improvements (Rosegrant et al, 2000), and to estimate the effect of hydrologic prediction uncertainties on water infrastructure decisions (Anghileri et al, 2016;Müller & Thompson, 2019). However, we argue that stronger integration between economics and hydrology will help both fields address two key methodological problems that emerge when studying human-water interactions in data-scarce and nonstationary environments:…”

“…Hydroeconomic models integrate the economic notions of scarcity and value by representing water demand as an explicit function of water availability rather than a fixed volumetric requirement (Harou et al, ). These insights allowed researchers to assess, for instance, the potential for markets to mitigate the effect of climate variability (Characklis et al, ), to evaluate the effectiveness of water use efficiency improvements (Rosegrant et al, ), and to estimate the effect of hydrologic prediction uncertainties on water infrastructure decisions (Anghileri et al, ; Müller & Thompson, ). However, we argue that stronger integration between economics and hydrology will help both fields address two key methodological problems that emerge when studying human‐water interactions in data‐scarce and nonstationary environments: Empirical causal inference: Causal relationships are challenging to distinguish from mere statistical correlations in empirical (data‐driven) research due to feedbacks and data limitations. Theoretical explanatory power: Inadequate theoretical treatment of dynamic feedbacks between humans and water gives rise to paradoxes that cannot be resolved by hydrology or economics alone. …”

Using Natural Experiments and Counterfactuals for Causal Assessment: River Salinity and the Ganges Water Agreement

“…In more complex scenarios, counterfactuals could be similarly constructed from observations using results from paired catchment studies (Brown et al, 2005), space-for-time substitution (Wagener et al, 2010), or large-sample hydrology (Gupta et al, 2014). However, observations of policy-relevant variables in hydrology are often challenging to obtain (see Hrachowitz et al, 2013), which motivates the use of models to bridge observation gaps (Müller & Thompson, 2019) and assess causal relationships (Beven, 2012).…”

Climatic and anthropogenic drivers of a drying Himalayan river

“…In perhaps the most common approach, watershed modeling is used to simulate streamflow, and the calibrated model is used to infer hydrological relationships and conduct the attribution (e.g., Liu et al, 2019). Goodness of fit and other model evaluation metrics influence which models or calibration parameters are given more credibility (Müller and Thompson, 2019) and which models are, in turn, used to identify the causal processes in the attribution. The major challenge of this approach for attribution is the difficulty in validating hydrological processes within the model.…”