A Resource Allocation Framework for Experiment-Based Validation of Numerical Models

Atamturktur, Sez; Hegenderfer, Joshua; Williams, Brian J.; Egeberg, Mark N.A; Lebensohn, Ricardo A.; Unal, Cetin

doi:10.1080/15376494.2013.828819

Cited by 22 publications

(18 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One of the best-known implementations of this approach is that of Kennedy and O'Hagan [2], where Gaussian Processes (GPs) were used to create statistical models of model discrepancy (as well as emulators, for the cases where simulations of the system of interest are computationally expensive). This idea has contributed to many works in the field of model validation, including generalised validation frameworks [4][5] [6], model validation metrics that consider model discrepancy [1], metrics to establish when sufficient experimental data has been gathered in the model validation process [7] and approaches for resource allocation (between code development and experimental testing [8] or the development of substructures within a model [9]).…”

Section: Literature Reviewmentioning

confidence: 99%

Towards the Validation of Dynamical Models in Regions where there is no Data

Green¹,

Chodora²,

Zhu³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

The creation of computer models is often driven by the need to make predictions in regions where there is no data (i.e. extrapolations). This makes validation challenging as it is difficult to ensure that a model will be suitable when it is applied in a region where there are no observations of the system of interest. The current paper proposes a method that can reveal flaws in a model which may be difficult to identify using traditional approaches for model calibration and validation. The method specifically targets the situation where one is attempting to model a dynamical system that is believed to possess time-invariant calibration parameters. The proposed approach allows these parameters to vary with time, even though it is believed that they are time-invariant. The of such an analysis is to identify key discrepancies - indications that a model has inherent flaws and, as a result, should not be used to influence decisions in regions where there is no data. The proposed method isn't necessarily a predictor of extrapolation performance, rather, it is a stringent test that, the authors believe, should be applied before extrapolation is attempted. The approach could therefore form a useful part of wider validation frameworks in the future.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Towards the Validation of Dynamical Models in Regions where there is no Data

Green¹,

Chodora²,

Zhu³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The metric is transformed to range between zero and infinity with zero representing the poorest possible coverage and infinity representing perfect coverage. This transformation is accomplished using the following functional form: The PMI has been established as a quantitative and objective metric to evaluate predictive capabilities of numerical models and has been applied to for instance the Preston-Tonks-Wallace model of plastic deformation [3], the Viscoplastic Self-Consistent (VPSC) material model [21], and the nuclear fuel performance code, LIFEIV [22]. Recently modified by Stull et al [9], the PMI includes four attributes: coverage of the domain,  c , robustness to model parameter uncertainty,  S , scaled discrepancy bias,  S , and model complexity, N K, as described in Eq.…”

Section: Transforming the Proposed Coverage Metric Into An Intuitive Indicatormentioning

confidence: 99%

“…The governing equation is written as [23]:  is a normalization factor [23]. A large number of parameters are required to completely describe the crystallographic textures using weights associated with a partition of 3-D orientation space [21]. However, for calibration and validation purposes, the final textures can be characterized by two components: (i) intensity associated with a retained (001) cube texture and (ii) intensity associated with a (101) compression texture.…”

Section: Vpsc Materials Modelmentioning

confidence: 99%

Defining coverage of an operational domain using a modified nearest-neighbor metric

Atamturktur

Egeberg

Hemez

et al. 2015

Mechanical Systems and Signal Processing

View full text Add to dashboard Cite

Validation experiments are conducted at discrete settings within an operational domain to assess the predictive maturity of a model that is ultimately used to predict over the entire operational domain. Unless this domain is sufficiently explored with validation experiments, satisfactory model performance at these discrete, tested settings would be insufficient to ensure satisfactory model performance throughout the entire operational domain. The goal of coverage metrics is then to reveal how well a set of validation experiments represents an operational domain. The authors identify the criteria of an exemplary coverage metric, evaluate the ability of existing coverage metrics to fulfill these criteria, and propose a new, improved coverage metric. The proposed metric favors interpolation over extrapolation through a penalty function, causing the metric to prefer a design of validation experiments near the boundaries of the domain, while simultaneously exploring inside the domain. Furthermore, the proposed metric allows the coverage to account for the relative influence of each dimension of the domain on the model output. Application of the proposed coverage metric on a practical, non-trivial two-dimensional problem is demonstrated on the Viscoplastic Self-Consistent material plasticity code for 5182 aluminum alloy. Furthermore, the proposed metric is compared to existing coverage metrics considering a high dimensional problem with application to the Rosenbrock function.

show abstract

“…The EIPS and EDIST are reportedly the most effective criteria for improving two specific attributes of the PMI metric: discrepancy and coverage (Atamturktur et al 2013). The VPSC code for the application of interest has 12 uncertain parameters, which are calibrated against the physical experiments on the 5182 aluminum alloy in Atamturktur et al (2014b). In this study, the synthetic experimental data for both the starting set of experiments and the BSD-selected experimental batches were generated using the posterior mean values of the input parameters (Table 2) reported in Atamturktur et al (2014b).…”

Section: Proof-of-concept Application: Vpsc Materials Modelmentioning

confidence: 99%

“…The VPSC code for the application of interest has 12 uncertain parameters, which are calibrated against the physical experiments on the 5182 aluminum alloy in Atamturktur et al (2014b). In this study, the synthetic experimental data for both the starting set of experiments and the BSD-selected experimental batches were generated using the posterior mean values of the input parameters (Table 2) reported in Atamturktur et al (2014b). The operational domain was defined by the range of control parameters, strain rate, and temperature given in Table 2.…”

Section: Proof-of-concept Application: Vpsc Materials Modelmentioning

confidence: 99%

Selection Criterion Based on an Exploration-Exploitation Approach for Optimal Design of Experiments

Atamturktur

Hegenderfer²,

Williams

et al. 2015

J. Eng. Mech.

View full text Add to dashboard Cite

Modeling and simulation are being relied upon in many fields of science and engineering as computational surrogates for experimental testing. To justify the use of these simulations for decision making, however, it is critical to determine, and when necessary mitigate, the biases and uncertainties in model predictions, a task that invariably requires validation experiments. To use experimental resources efficiently, validation experiments must be designed to achieve the maximum possible increases in model predictive ability through the calibration of the model against experiments. This need for efficiency is addressed by the concept of optimally designing validation experiments, which constitutes optimizing a predefined criterion while selecting the settings of experiments. This paper presents an improved optimization criterion that incorporates two important factors for the optimal design of validation experiments: (1) how well the model reproduces the validation experiments, and (2) how well the validation experiments cover the domain of applicability. The criterion presented herein selects the appropriate settings for future experiments with the goal of achieving a desired level of predictive ability in the computer model through the use of a minimal number of validation experiments. The criterion explores the entirety of the application domain by including the effect of coverage, and exploits areas of the domain with high variability by including the effect of empirically defined discrepancy bias. The effectiveness of this new criterion is compared with two well-established criteria through a simulated case study involving the stress-strain response and textural evolution of polycrystalline materials. The proposed criterion is demonstrated as efficient at improving the predictive capabilities of the numerical model, particularly when the amount of experimental data available for validation is low.

show abstract

A Resource Allocation Framework for Experiment-Based Validation of Numerical Models

Cited by 22 publications

References 33 publications

Towards the Validation of Dynamical Models in Regions where there is no Data

Towards the Validation of Dynamical Models in Regions where there is no Data

Defining coverage of an operational domain using a modified nearest-neighbor metric

Selection Criterion Based on an Exploration-Exploitation Approach for Optimal Design of Experiments

Contact Info

Product

Resources

About

A Resource Allocation Framework for Experiment-Based Validation of Numerical Models

Cited by 22 publications

References 33 publications

Towards the Validation of Dynamical Models in Regions where there is no&nbsp;Data

Towards the Validation of Dynamical Models in Regions where there is no&nbsp;Data

Defining coverage of an operational domain using a modified nearest-neighbor metric

Selection Criterion Based on an Exploration-Exploitation Approach for Optimal Design of Experiments

Contact Info

Product

Resources

About

Towards the Validation of Dynamical Models in Regions where there is no Data

Towards the Validation of Dynamical Models in Regions where there is no Data