Verification, validation, and predictive capability in computational engineering and physics

Abstract

Developers of computer codes, analysts who use the codes, and decision makers who rely on the results of the analyses face a critical question: How should confidence in modeling and simulation be critically assessed? Verification and validation (V&V) of computational simulations are the primary methods for building and quantifying this confidence. Briefly, verification is the assessment of the accuracy of the solution to a computational model. Validation is the assessment of the accuracy of a computational simulation by comparison with experimental data. In verification, the relationship of the simulation to the real world is not an issue. In validation, the relationship between computation and the real world, i.e., experimental data, is the issue.-3 -This paper presents our viewpoint of the state of the art in V&V in computational physics. (In this paper we refer to all fields of computational engineering and physics, e.g., computational fluid dynamics, computational solid mechanics, structural dynamics, shock wave physics, computational chemistry, etc., as computational physics.) We do not provide a comprehensive review of the multitudinous contributions to V&V, although we do reference a large number of previous works from many fields. We have attempted to bring together many different perspectives on V&V, highlight those perspectives that are effective from a practical engineering viewpoint, suggest future research topics, and discuss key implementation issues that are necessary to improve the effectiveness of V&V. We describe our view of the framework in which predictive capability relies on V&V, as well as other factors that affect predictive capability. Our opinions about the research needs and management issues in V&V are very practical: What methods and techniques need to be developed and what changes in the views of management need to occur to increase the usefulness, reliability, and impact of computational physics for decision making about engineering systems?We review the state of the art in V&V over a wide range of topics, for example, prioritization of V&V activities using the Phenomena Identification and Ranking Table (PIRT), code verification, software quality assurance (SQA), numerical error estimation, hierarchical experiments for validation, characteristics of validation experiments, the need to perform nondeterministic computational simulations in comparisons with experimental data, and validation metrics. We then provide an extensive discussion of V&V research and implementation issues that we believe must be addressed for V&V to be more effective in improving confidence in computational predictive capability. Some of the research topics addressed are development of improved procedures for the use of the PIRT for prioritizing V&V activities, the method of manufactured solutions for code verification, development and use of hierarchical validation diagrams, and the construction and use of validation metrics incorporating statistical measures. Some of the implementation topics address...

Validation metrics for turbulent plasma transport

“…In this paper, the definitions given by Terry et al 14 will be used. As noted above, alternative definitions and terminology discussions can be found in professional association guidelines, 1,5 journal reviews, [6][7][8][9][10][11] and textbooks. 12,13 A summary of simple "plain English" descriptions (not definitions) of the key concepts is provided in Table I.…”

“…This formal quantification of model accuracy, as opposed to more subjective assessments of model fidelity often found in the literature such as "good agreement," "qualitatively consistent," or "clearly differs," is what distinguishes validation as an activity distinct from many earlier model-experiment comparison studies. Indeed, Oberkampf et al 9 state that "[t]he specification and use of validation metrics comprises the most important practice in validation studies. "…”

“…Given the importance of rigorous validation studies in many fields, it is no surprise that there is now a broad and well-established literature in the field, ranging from guidelines and best practices prescribed by professional societies 1,5 to journal reviews [6][7][8][9][10][11] and textbooks, 12,13 in addition to the numerous articles and reports detailing the outcomes of individual studies. In the area of plasma physics and fusion energy research, key review articles include those by Terry et al 14 and Greenwald.…”

Effect of delayed link failure on probability of loss of assured safety in temperature-dependent systems with multiple weak and strong links

“…[16][17][18][19][20][21][22][23][24][25] Consistent with this importance, a procedure for verifying the conceptual development and numerical implementation of integration algorithms used to determine pF with temperature-dependent delays is now described. This procedure also provides the basis for an alternative approach to the numerical approximation of pF.…”

Benchmark numerical simulations of rarefied non-reacting gas flows using an open-source DSMC code

“…Computational codes can only be considered reliable if they pass a through rigorous process of verification and validation (V&V). In an effort to standardize the V&V process, a significant 5 amount of literature has been produced on the subject, e.g., [1][2][3][4][5][6][7][8]. The present study adopts the V&V definition stated in Ref.…”