The Method of Manufactured Universes for validating uncertainty quantification methods

Stripling, Hayes F.; Adams, Marvin L.; McClarren, Ryan G.; Mallick, Bani K.

doi:10.1016/j.ress.2010.11.012

Cited by 21 publications

(22 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since determination of the "true" value in nature is challenging (e.g., see McCroskey [12]), the recentlyproposed Method of Manufactured Universes (MMU) developed by Stripling et al [13] is used to assess the different model validation techniques in an unambiguous fashion. Similar to the method of manufactured solutions used for code verification, MMU involves the generation of a manufactured reality, or "universe", from which experimental observations can be made (possibly including experimental measurements errors).…”

Section: Methods Of Manufactured Universesmentioning

confidence: 99%

“…Since the true behavior of "reality" is known in this manufactured universe, the exact evaluation of modeling errors in the lower-fidelity model can be performed. Stripling et al [13] suggest that the manufactured reality be based on a high-fidelity model so as to obtain similar qualitative behavior as that found in nature. This approach can be used to compare and contrast different approaches for estimating model form uncertainty in the presence of random experimental measurement error, experimental bias errors, modeling errors, and uncertainties (both random and those due to lack of knowledge) in the model inputs.…”

Section: Methods Of Manufactured Universesmentioning

confidence: 99%

See 1 more Smart Citation

Evaluation of Model Validation Techniques in the Presence of Uncertainty

Voyles

Roy

2014

16th AIAA Non-Deterministic Approaches Conference

View full text Add to dashboard Cite

Model validation is the assessment of the correctness of a model relative to experimental data. The results of a model validation study can be used to either quantify the model form uncertainty or to improve the model (i.e., calibration). The model validation process is generally complicated by the fact that both the simulation outcomes and the experimental outcomes include significant uncertainty which can come in the form of random (aleatory) uncertainties, lack-of-knowledge (epistemic) uncertainties, and bias errors. In this paper, we examine four different approaches to model validation: 1) the area validation metric, 2) the standard validation uncertainty, 3) a modified area validation metric, and 4) an approach based on comparison on probability density functions. In order to provide a rigorous assessment of these model validation frameworks, we employ the recently developed Method of Manufactured Universes (MMU). The main advantage of MMU is that the "true" value in nature is known, thus allowing a rigorous assessment of whether the model validation approaches provide conservative and tight bounds on the true model error. MMU is applied to the compressible turbulent flow over a NACA 0012 airfoil. The "true" values in the manufactured universe are found using turbulent computational fluid dynamics simulations while the "model" employs simplified lift and drag estimates based on thin airfoil theory and empirical drag correlations. Preliminary results indicate that the area validation metrics are less susceptible to over-predict model form uncertainty and that the modified area validation metric further tightens the bounds on the true model error relative to the original area validation metric.

show abstract

Section: Methods Of Manufactured Universesmentioning

confidence: 99%

Section: Methods Of Manufactured Universesmentioning

confidence: 99%

Evaluation of Model Validation Techniques in the Presence of Uncertainty

Voyles

Roy

2014

16th AIAA Non-Deterministic Approaches Conference

View full text Add to dashboard Cite

show abstract

“…In our fourth example, we illustrate the framework for a particle transport model quantifying angular flux in a 1-D slab. The purpose of this example is to illustrate the similarities between our high-to-low framework and the Method of Manufactured Universes [19].…”

Section: Examplesmentioning

confidence: 99%

“…The use of high-fidelity simulation models to calibrate lower-fidelity models is closely related to the Method of Manufactured Universes (MMU) proposed in [19]. In this framework, the researcher defines the laws of their manufactured "universe" and simulates experimental data-coinciding with the high-fidelity synthetic data in this investigation-that follows these laws and may contain measurement error.…”

Section: Introductionmentioning

confidence: 99%

An information theoretic approach to use high-fidelity codes to calibrate low-fidelity codes

Lewis

Smith

Williams

et al. 2016

Journal of Computational Physics

View full text Add to dashboard Cite

For many simulation models, it can be prohibitively expensive or physically infeasible to obtain a complete set of experimental data to calibrate model parameters. In such cases, one can alternatively employ validated higherfidelity codes to generate simulated data, which can be used to calibrate the lower-fidelity code. In this paper, we employ an information-theoretic framework to determine the reduction in parameter uncertainty that is obtained by evaluating the high-fidelity code at a specific set of design conditions. These conditions are chosen sequentially, based on the the amount of information that they contribute to the low-fidelity model parameters. The goal is to employ Bayesian experimental design techniques to minimize the number of high-fidelity code evaluations required to accurately calibrate the low-fidelity model. We illustrate the performance of this framework using heat and diffusion examples, a 1-D kinetic neutron diffusion equation, and a particle transport model, and include initial results from the integration of the high-fidelity thermal-hydraulics code Hydra-TH with a low-fidelity exponential model for the friction correlation factor.

show abstract

“…To rigorously assess the performance of the validation metrics considered in this paper, a manufactured universe [11] is implemented to provide an artificial "nature" (the manufactured universe) where the true value in nature can be known. Experiments can then be performed on the manufactured universe and can include random errors (achieved though Monte Carlo sampling) and bias errors.…”

Section: A True Value In Naturementioning

confidence: 99%

Evaluation of Model Validation Techniques in the Presence of Aleatory and Epistemic Input Uncertainties

Voyles

Roy

2015

17th AIAA Non-Deterministic Approaches Conference

View full text Add to dashboard Cite

Model validation is the assessment of the "correctness" of a given model relative to experimental data. The results of a model validation study can be used to quantify the model form uncertainty, to select between different models, or to improve the model (i.e., through calibration or model updating). The process of model validation is complicated by the fact that both the simulation and experimental outcomes include significant uncertainty, which can come in the form of aleatory (random) uncertainties, epistemic (lack-of-knowledge) uncertainties, and bias errors. The application of Probability Bounds Analysis for treating mixed (i.e., containing both aleatory and epistemic) input uncertainties results in a family of cumulative distribution functions (CDFs) of the simulation outcomes, which is referred to as a "probability box" or "p-box." The fact that a family of CDFs, as opposed to a single CDF, is required to characterize the outcomes complicates the implementation of model validation techniques. In this paper, we will examine the following approaches to model validation and assess their ability to handle problems with mixed uncertainties: 1) the area validation metric, 2) a modified area validation metric with a factor of safety, 3) a modified area validation metric with confidence intervals, 4) the standard validation uncertainty, and 5) the difference in simulation and experimental means. To provide a rigorous assessment of these model validation techniques, we employ the recently developed Method of Manufactured Universes (MMU), where "true values in nature" are constructed by the analyst to reflect the behavior of a physical reality of interest. Here, MMU is applied to the compressible turbulent flow over a NACA 0012 airfoil, where the "true" values are constructed using turbulent computational fluid dynamics simulations while the "model" employs simplified lift and drag estimates based on thin airfoil theory and empirical lift and drag correlations. Preliminary results suggest that the modified area validation metric implementations provide more conservative estimates of the model form uncertainty than the area validation metric for mean values, with the confidence-interval implementation also returning smaller uncertainty ratios.

show abstract

The Method of Manufactured Universes for validating uncertainty quantification methods

Cited by 21 publications

References 14 publications

Evaluation of Model Validation Techniques in the Presence of Uncertainty

Evaluation of Model Validation Techniques in the Presence of Uncertainty

An information theoretic approach to use high-fidelity codes to calibrate low-fidelity codes

Evaluation of Model Validation Techniques in the Presence of Aleatory and Epistemic Input Uncertainties

Contact Info

Product

Resources

About