A General Strategy for Physics-Based Model Validation Illustrated with Earthquake Phenomenology, Atmospheric Radiative Transfer, and Computational Fluid Dynamics

Abstract

Summary. Validation is often defined as the process of determining the degree to which a model is an accurate representation of the real world from the perspective of its intended uses. Validation is crucial as industries and governments depend increasingly on predictions by computer models to justify their decisions. In this article, we survey the model validation literature and propose to formulate validation as an iterative construction process that mimics the process occurring implicitly in the minds of scientists. We thus offer a formal representation of the progressive build-up of trust in the model, and thereby replace incapacitating claims on the impossibility of validating a given model by an adaptive process of constructive approximation. This approach is better adapted to the fuzzy, coarse-grained nature of validation. Our procedure factors in the degree of redundancy versus novelty of the experiments used for validation as well as the degree to which the model predicts the observations. We illustrate the new methodology first with the maturation of Quantum Mechanics as the arguably best established physics theory and then with several concrete examples drawn from some of our primary scientific interests: a cellular automaton model for earthquakes, an anomalous diffusion model for solar radiation transport in the cloudy atmosphere, and a computational fluid dynamics code for the Richtmyer-Meshkov instability.

Defining predictive maturity for validated numerical simulations

“…[7,8]. Predictive maturity, as described in the above paragraph, comes with the drawback of relying on qualitative statements and expert judgment, even if the underlying V&V activities are quantitative in nature.…”

“…[7,8], the authors focus on the development of a goodness-of-fit metric to quantify the agreement, or correlation, between predictions and measurements. Their proposal includes the concept of ''novelty" of a piece of information or experimental measurement.…”

Selection Criterion Based on an Exploration-Exploitation Approach for Optimal Design of Experiments

“…To effectively compare the performance of multiple BSD criteria, a quantitative and objective metric for capturing a model's predictive capability throughout a prescribed operational domain is necessary. Recently, many institutions have led efforts to develop such metrics that aim to assess the predictive capability of complex numerical models (Harmon and Youngblood 2005;Sornette et al 2006; National Aeronautics and Space Administration (NASA) 2007; Oberkampf et al 2007;Sornette et al 2007;Zang 2008;Hemez et al 2010). Hemez et al (2010) provided a detailed review of the literature on predictive capability metrics.…”

Conditions for Quantum Interference in Cognitive Sciences

“…In other words, the process of validation must be developed, which consists in making novel predictions to empirical situations that have not been previously used to motivate the theory. Fitting a model to empirical observations can serve only as a first trivial step in the whole chain of model validation (Sornette, Davis, Ide, Vixie, Pisarenko, & Kamm, 2007;Sornette, Davis, Kamm, & Ide, 2008).…”