A statistical concept to assess the uncertainty in Bayesian model weights and its impact on model ranking

Schöniger, Anneli; Wöhling, Thomas; Nowak, Wolfgang

doi:10.1002/2015wr016918

Cited by 35 publications

(27 citation statements)

References 68 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The problem of segregating these three sources of uncertainty has been studied extensively (e.g., Keenan et al 2012;Montanari and Koutsoyiannis 2012;Schöniger et al 2015;Liu and Gupta 2007;Kavetski et al 2006;Draper 1995;Oberkampf et al 2002;Wilby and Harris 2006;Poulin et al 2011;Clark et al 2011). Almost ubiquitously, the methods that have been applied to this problem are based on the chain rule of probability theory (Liu and Gupta 2007).…”

Section: Introductionmentioning

confidence: 99%

Benchmarking NLDAS-2 Soil Moisture and Evapotranspiration to Separate Uncertainty Contributions

Nearing

Mocko

Peters‐Lidard

et al. 2016

Journal of Hydrometeorology

View full text Add to dashboard Cite

Model benchmarking allows us to separate uncertainty in model predictions caused by model inputs from uncertainty due to model structural error. We extend this method with a "large-sample" approach (using data from multiple field sites) to measure prediction uncertainty caused by errors in (i) forcing data, (ii) model parameters, and (iii) model structure, and use it to compare the efficiency of soil moisture state and evapotranspiration flux predictions made by the four land surface models in the North American Land Data Assimilation System Phase 2 (NLDAS-2). Parameters dominated uncertainty in soil moisture estimates and forcing data dominated uncertainty in evapotranspiration estimates; however, the models themselves used only a fraction of the information available to them. This means that there is significant potential to improve all three components of the NLDAS-2 system. In particular, continued work toward refining the parameter maps and look-up tables, the forcing data measurement and processing, and also the land surface models themselves, has potential to result in improved estimates of surface mass and energy balances.

show abstract

Section: Introductionmentioning

confidence: 99%

Benchmarking NLDAS-2 Soil Moisture and Evapotranspiration to Separate Uncertainty Contributions

Nearing

Mocko

Peters‐Lidard

et al. 2016

Journal of Hydrometeorology

View full text Add to dashboard Cite

show abstract

“…As the magnitude and number of unknown unknowns in the model is unknown, there is no way of determining the value of σ. The robustness of model ranking as defined by [48], was evaluated in a sensitivity analysis by varying the standard deviation of the relative error of the water balance between 0% and 10%. To avoid making wrong model selection decisions based on the results, we selected a standard deviation of the model error that is least decisive.…”

Section: Water Balance Modelmentioning

confidence: 99%

Hydrogeological Bayesian Hypothesis Testing through Trans-Dimensional Sampling of a Stochastic Water Balance Model

Enemark

Peeters

Mallants

et al. 2019

Water

View full text Add to dashboard Cite

Conceptual uncertainty is considered one of the major sources of uncertainty in groundwater flow modelling. In this regard, hypothesis testing is essential to increase system understanding by refuting alternative conceptual models. Often a stepwise approach, with respect to complexity, is promoted but hypothesis testing of simple groundwater models is rarely applied. We present an approach to model-based Bayesian hypothesis testing in a simple groundwater balance model, which involves optimization of a model in function of both parameter values and conceptual model through trans-dimensional sampling. We apply the methodology to the Wildman River area, Northern Territory, Australia, where we set up 32 different conceptual models. A factorial approach to conceptual model development allows for direct attribution of differences in performance to individual uncertain components of the conceptual model. The method provides a screening tool for prioritizing research efforts while also giving more confidence to the predicted water balance compared to a deterministic water balance solution. We show that the testing of alternative conceptual models can be done efficiently with a simple additive and linear groundwater balance model and is best done relatively early in the groundwater modelling workflow.One branch of hypothesis testing is based on the Bayesian probability theory. In Bayesian hypothesis testing, a prior belief about the suitability of a conceptual model is updated to a posterior belief by evaluating the model performance against data. The performance of alternative models are then compared in order to quantitatively rank and potentially reject hypotheses based on the so-called Bayes factor [9,10]. Fields of application in hydrogeology include groundwater modelling [11,12], hydrogeophysics [13,14] and solute transport modelling [15,16]. In many applications, the data is not sufficient to allow for discrimination between the models, in which case Bayesian model averaging is often applied where model predictions are weighed according to their performance against data [17].In hydrogeology, conceptual models are often tested in mathematical models (e.g., [18,19]). Since a model comprises the description of a system as a whole, all assumptions and the interaction of assumptions are tested at once. Also, data that does not directly relate to the conceptually uncertain feature can be integrated because of the holistic testing of the system.A stepwise approach in regards to the complexity of groundwater flow modelling and hypothesis testing is often promoted [20,21]. In this paper the simplicity of a model is defined in terms of setup and run-time. In a stepwise approach, complexity is gradually built up, and involves testing the models in each step to better understand the relative importance of various assumptions. This is opposed to starting with a complex model where all known processes and structural aspects are incorporated "because they exist, not because they matter" [20].Although there seems to be a consensu...

show abstract

“…Note that while our study features a first-time implementation of PreDIA for Bayesian model weights, this algorithm is similar to the random sampling of measurement error for analyzing the robustness of BMA results, as presented in Schöniger et al [106]. A related schematic illustration of the implementation scheme can be found there.…”

Section: Conflicts Of Interestmentioning

confidence: 99%

Entropy-Based Experimental Design for Optimal Model Discrimination in the Geosciences

Nowak

Guthke

2016

Entropy

Self Cite

View full text Add to dashboard Cite

Abstract:Choosing between competing models lies at the heart of scientific work, and is a frequent motivation for experimentation. Optimal experimental design (OD) methods maximize the benefit of experiments towards a specified goal. We advance and demonstrate an OD approach to maximize the information gained towards model selection. We make use of so-called model choice indicators, which are random variables with an expected value equal to Bayesian model weights. Their uncertainty can be measured with Shannon entropy. Since the experimental data are still random variables in the planning phase of an experiment, we use mutual information (the expected reduction in Shannon entropy) to quantify the information gained from a proposed experimental design. For implementation, we use the Preposterior Data Impact Assessor framework (PreDIA), because it is free of the lower-order approximations of mutual information often found in the geosciences. In comparison to other studies in statistics, our framework is not restricted to sequential design or to discrete-valued data, and it can handle measurement errors. As an application example, we optimize an experiment about the transport of contaminants in clay, featuring the problem of choosing between competing isotherms to describe sorption. We compare the results of optimizing towards maximum model discrimination with an alternative OD approach that minimizes the overall predictive uncertainty under model choice uncertainty.

show abstract

A statistical concept to assess the uncertainty in Bayesian model weights and its impact on model ranking

Cited by 35 publications

References 68 publications

Benchmarking NLDAS-2 Soil Moisture and Evapotranspiration to Separate Uncertainty Contributions

Benchmarking NLDAS-2 Soil Moisture and Evapotranspiration to Separate Uncertainty Contributions

Hydrogeological Bayesian Hypothesis Testing through Trans-Dimensional Sampling of a Stochastic Water Balance Model

Entropy-Based Experimental Design for Optimal Model Discrimination in the Geosciences

Contact Info

Product

Resources

About