Towards a Credibility Assessment of Models and Simulations

Blattnig, Steve R.; Green, Lawrence; Luckring, James M.; Morrison, Joseph H.; Tripathi, R. K.; Zang, Thomas A.

doi:10.2514/6.2008-2156

Cited by 17 publications

(17 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, that seems to defeat the real purpose of model validation. Recently, it has been recommended that assessments of model credibility be evaluated independent of requirements [1][2][3]47]. Stated another way, validation metrics should not be tied to notions of the adequacy of agreement between the model and experiment [1].…”

Section: Model Validationmentioning

confidence: 99%

“…Therefore, models will be relied upon to make decisions regarding vehicle design and overall mission planning. An increased dependence on model results makes it critical to effectively validate the models used in decision making in order to determine their accuracy and reliability [1][2][3][4]. To ensure confidence in the validation assessment, reliable and repeatable processes must be implemented.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Validation of nuclear models used in space radiation shielding applications

Norman

Blattnig

2013

Journal of Computational Physics

View full text Add to dashboard Cite

Section: Model Validationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Validation of nuclear models used in space radiation shielding applications

Norman

Blattnig

2013

Journal of Computational Physics

View full text Add to dashboard Cite

“…Assessing credibility of modeling and simulation (M&S) results is a problem that has been well thought out in organizations like NASA, where they may directly affect human safety and mission success [ Blattni et al ., ]. Following the Columbia shuttle accident, NASA developed the technical standard NASA‐STD‐7009, and the subsequent handbook NASA‐HDBK‐7009, in order “to ensure that the credibility of the results from M&S is properly conveyed to those making critical decisions” [ National Aeronautics and Space Administration , ].…”

Section: Introductionmentioning

confidence: 99%

LIVVkit: An extensible, python‐based, land ice verification and validation toolkit for ice sheet models

Kennedy

Bennett

Evans

et al. 2017

J Adv Model Earth Syst

View full text Add to dashboard Cite

To address the pressing need to better understand the behavior and complex interaction of ice sheets within the global Earth system, significant development of continental‐scale, dynamical ice sheet models is underway. Concurrent to the development of the Community Ice Sheet Model (CISM), the corresponding verification and validation (V&V) process is being coordinated through a new, robust, Python‐based extensible software package, the Land Ice Verification and Validation toolkit (LIVVkit). Incorporated into the typical ice sheet model development cycle, it provides robust and automated numerical verification, software verification, performance validation, and physical validation analyses on a variety of platforms, from personal laptops to the largest supercomputers. LIVVkit operates on sets of regression test and reference data sets, and provides comparisons for a suite of community prioritized tests, including configuration and parameter variations, bit‐for‐bit evaluation, and plots of model variables to indicate where differences occur. LIVVkit also provides an easily extensible framework to incorporate and analyze results of new intercomparison projects, new observation data, and new computing platforms. LIVVkit is designed for quick adaptation to additional ice sheet models via abstraction of model specific code, functions, and configurations into an ice sheet model description bundle outside the main LIVVkit structure. Ultimately, through shareable and accessible analysis output, LIVVkit is intended to help developers build confidence in their models and enhance the credibility of ice sheet models overall.

show abstract

“…A companion paper in this session discusses how the rigor that is associated with several key aspects of an M&S effort (including the UQ and UM practices) influences the perceived credibility of that M&S in a given situation. 14 Uncertainties in the technical performance metrics, along with those that are associated with schedule and budget, obviously introduce risk within any decision-making process. However, for decisions that are based upon M&S, the potential exists that the uncertainties (particularly of the technical performance metrics) may be significantly underestimated due to a wide range of factors; this potential for underestimated M&S uncertainty introduces additional risks that are distinct and that are typically not considered within NASA's decision-making processes.…”

Section: Caibmentioning

confidence: 99%

“…), by the DMSO 9 (now the Modeling and Simulation Coordination Office), and by NASA at Langley Research Center (LaRC) [14][15] and Marshall Space Flight Center. 16 A future effort will likely be made to compare and contrast the various ways these groups have chosen to approach the ideas of UQ and UM within decision-making processes.…”

Section: Caibmentioning

confidence: 99%

An Uncertainty Structure Matrix for Models and Simulations

Green¹,

Blattnig²,

Hamsch³

et al. 2008

49th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference &Amp;lt;br> 16th AIAA/ASME/AHS Ada

Self Cite

View full text Add to dashboard Cite

Software that is used for aerospace flight control and to display information to pilots and crew is expected to be correct and credible at all times. This type of software is typically developed under strict management processes, which are intended to reduce defects in the software product. However, modeling and simulation (M&S) software may exhibit varying degrees of correctness and credibility, depending on a large and complex set of factors. These factors include its intended use, the known physics and numerical approximations within the M&S, and the referent data set against which the M&S correctness is compared. The correctness and credibility of an M&S effort is closely correlated to the uncertainty management (UM) practices that are applied to the M&S effort. This paper describes an uncertainty structure matrix for M&S, which provides a set of objective descriptions for the possible states of UM practices within a given M&S effort. The columns in the uncertainty structure matrix contain UM elements or practices that are common across most M&S efforts, and the rows describe the potential levels of achievement in each of the elements. A practitioner can quickly look at the matrix to determine where an M&S effort falls based on a common set of UM practices that are described in absolute terms that can be applied to virtually any M&S effort. The matrix can also be used to plan those steps and resources that would be needed to improve the UM practices for a given M&S effort.

show abstract

Towards a Credibility Assessment of Models and Simulations

Cited by 17 publications

References 10 publications

Validation of nuclear models used in space radiation shielding applications

Validation of nuclear models used in space radiation shielding applications

LIVVkit: An extensible, python‐based, land ice verification and validation toolkit for ice sheet models

An Uncertainty Structure Matrix for Models and Simulations

Contact Info

Product

Resources

About