A probabilistic metric for the validation of computational models

Abstract

A new validation metric is proposed that combines the use of a threshold based on the uncertainty in the measurement data with a normalized relative error, and that is robust in the presence of large variations in the data. The outcome from the metric is the probability that a model's predictions are representative of the real world based on the specific conditions and confidence level pertaining to the experiment from which the measurements were acquired. Relative error metrics are traditionally designed for use with a series of data values, but orthogonal decomposition has been employed to reduce the dimensionality of data matrices to feature vectors so that the metric can be applied to fields of data. Three previously published case studies are employed to demonstrate the efficacy of this quantitative approach to the validation process in the discipline of structural analysis, for which historical data were available; however, the concept could be applied to a wide range of disciplines and sectors where modelling and simulation play a pivotal role.

The validation of a full-field deformation analysis of an aircraft panel: A case study

“…The methodologies and the steps described in this paper demonstrate a successful translation of laboratory research into the industrial environment. The validation process that was implemented in this aerospace case study is based on recently published CEN Workshop Agreement (CWA 16799:2014) 2 with addition of a probabilistic validation metric 14 that allows the relative difference between fields of data to be Figure 5. Plots comparing measured and predicted displacement fields in Figure 2 using the approach described in CWA16779 which involves plotting against one another the coefficients of the feature vectors representing the data fields from the simulation (yaxes) and experiment (x-axes) together with lines representing equality (solid) and equality plus and minus the expanded uncertainty in the measurements (dashed lines).…”

“…The methodologies and the steps described in this paper demonstrate a successful translation of laboratory research into the industrial environment. The validation process that was implemented in this aerospace case study is based on recently published CEN Workshop Agreement (CWA 16799:2014) 2 with addition of a probabilistic validation metric 14 that allows the relative difference between fields of data to be assessed against their uncertainty. These methodologies and the sequence of steps that constituted the validation process can be employed in other industrial sectors where data can be treated as two-dimensional fields or images and the simulations are used for critical decision making.…”

“…Dvurecenska et al 14 have recently proposed a validation metric that extends the CEN methodology and quantifies the similarities between predicted and measured responses. The metric is based on evaluating relative errors, e k between the components of two feature vectors against a threshold, u threshold such that…”

“…Instead of a visual, essentially qualitative and subjective, side-by-side comparison of such deformation data, it has been previously proposed to utilise image decomposition to process the data from both experiments and simulations to support model updating 13 and later for validation. 10,14 By treating a data field, for example a displacement field, as an image in which the magnitude is represented by a colour or grey level value, image decomposition techniques can be easily applied to both simulation and experimental outputs. Image decomposition is a form of orthogonal decomposition based on the principle of fitting a set of selected orthogonal polynomials to an image and is commonly used to both compress data and to identify shapes or features in images for applications such as object tracking, face and natural structures recognition.…”

The role of validation in establishing the scientific credibility of predictive toxicology approaches intended for regulatory application

“… 41] should be used but when this is not viable then some form of qualitative concordance should be undertaken. In the extreme, it is even possible to provide a relative error for the prediction expressed as a probability that the prediction belongs to the same population as measured data [42] . However, it is more likely that the intrinsic uncontrollability of real-world biological phenomena and the difficulty of capturing the complexity of the biological system will limit the quantity and quality of data available for comparison with predictions.…”