Evaluation of Model Validation Techniques in the Presence of Aleatory and Epistemic Input Uncertainties

Abstract

Model validation is the assessment of the "correctness" of a given model relative to experimental data. The results of a model validation study can be used to quantify the model form uncertainty, to select between different models, or to improve the model (i.e., through calibration or model updating). The process of model validation is complicated by the fact that both the simulation and experimental outcomes include significant uncertainty, which can come in the form of aleatory (random) uncertainties, epistemic (lack-of-knowledge) uncertainties, and bias errors. The application of Probability Bounds Analysis for treating mixed (i.e., containing both aleatory and epistemic) input uncertainties results in a family of cumulative distribution functions (CDFs) of the simulation outcomes, which is referred to as a "probability box" or "p-box." The fact that a family of CDFs, as opposed to a single CDF, is required to characterize the outcomes complicates the implementation of model validation techniques. In this paper, we will examine the following approaches to model validation and assess their ability to handle problems with mixed uncertainties: 1) the area validation metric, 2) a modified area validation metric with a factor of safety, 3) a modified area validation metric with confidence intervals, 4) the standard validation uncertainty, and 5) the difference in simulation and experimental means. To provide a rigorous assessment of these model validation techniques, we employ the recently developed Method of Manufactured Universes (MMU), where "true values in nature" are constructed by the analyst to reflect the behavior of a physical reality of interest. Here, MMU is applied to the compressible turbulent flow over a NACA 0012 airfoil, where the "true" values are constructed using turbulent computational fluid dynamics simulations while the "model" employs simplified lift and drag estimates based on thin airfoil theory and empirical lift and drag correlations. Preliminary results suggest that the modified area validation metric implementations provide more conservative estimates of the model form uncertainty than the area validation metric for mean values, with the confidence-interval implementation also returning smaller uncertainty ratios.

Propagation of Input Uncertainty in Presence of Model-Form Uncertainty: A Multifidelity Approach for Computational Fluid Dynamics Applications

“…In this proof-of-concept study we only conduct approximate comparison via visual observations, which can be inadequate when the qualities of two predicted distributions are not obvious. In those cases, more advanced metrics for comparing probability distributions are required, e.g., Area Validation Metric [25] and its modified variant [26]. Figure 4 shows that the normally distributed input uncertainty is mapped to a bi-modal distribution in the output uncertainty distribution.…”

Probability Bounds Analysis Applied to the Sandia Verification and Validation Challenge Problem

“…4.6). We then use the MAVM to estimate model form uncertainty in von Mises stress [23,24]. Since this uncertainty is epistemic, we treat it as an interval about the simulation outcome.…”

“…jFðYÞ À S n ðYÞjdY (6) where FðYÞ is the p-box from the simulations, S n ðYÞ is the empirical CDF from the experiments, and Y is the SRQ of interest. The MAVM employed here [24] accounts for regions in the cumulative probability space where the experimental values are larger than the simulation values (d þ ) and are smaller than the simulation values (d À ) (see Refs. [23,24] for a complete discussion on the process of MAVM evaluation).…”

“…The MAVM employed here [24] accounts for regions in the cumulative probability space where the experimental values are larger than the simulation values (d þ ) and are smaller than the simulation values (d À ) (see Refs. [23,24] for a complete discussion on the process of MAVM evaluation). Once these metrics are computed, the model form uncertainty is constructed as the following interval around the simulation p-box:…”

“…where k is the number of available experimental samples, F 1 ¼ 1:25, and F 0 ¼ 4:0. For additional conservativeness, the "average area" definition given by Voyles and Roy [23,24] is used. Note that another formulation for F s can be found in Ref.…”