A new specification-based qualitative metric for simulation model validity

Abstract

Informal validation techniques such as simulation are extensively used in the development of embedded systems. Formal approaches such as model-checking and testing are important means to carry out Verification and Validation (V&V) activities. Model-checking consists in exploring all possible behaviours of a model in order to perform a qualitative and quantitative analysis. However, this method remains of limited use as it runs into the problem of combinatorial explosion. Testing and model-checking do not take into account the context of use objectives of the model. Simulation overcomes these problems but it is not exhaustive. Submitted to simulation scenarios which are an operational formulation of the V&V activity considered, simulation consists in exploring a subset of the state space of the model. This paper proposes a formal approach to assess simulation scenarios. The formal specification of a model and the simulation scenarios applied to that model serve to compute the effective evolutions taken by the simulation. It is then possible to check whether a simulation fulfils its intended purpose. To illustrate this approach, the application study of an intelligent cruise controller is presented. The main contribution of this paper is that combining simulation objectives and formal methods leads to define a qualitative metric for a simulation evaluation without running a simulation.

A Methodological Approach to Assessment and Reporting of the Model Adequacy in Simulation Studies

“…The US National Research Council report depicts model evaluation as "the process of deciding whether and when a model is suitable for its intended purpose" (Council, 2007). Though there are a variety of methods for evaluation of model adequacy, there still exists a requirement of a testing procedure which considers both the quantitative and qualitative methodology (Foures et al, 2016;Denil et al, 2017). Also, a step by step methodology will help the future researchers while assessing the model adequacy of the simulation model in the scientific environment.…”

“…One of the most widely used methods is quantitative methods for verification and validation. However, a recent study suggests that to explore all possible behaviours of models one should take into account both qualitative and quantitative approach (Foures et al, 2016). Another point to be considered is that model is a representation of process or phenomenon.…”

“…There has been strong criticism of the concept of model validation and verification (Addiscott, Smith, & Bradbury, 1995). It is partly due to the philosophy that it is impossible to validate all components of the model (Oreskes, Shrader-Frechette, & Belitz, 1994;Saltelli, Tarantola, Campolongo, & Ratto, 2004;Foures, Albert, & Nketsa, 2016). There are formal approaches for model verification and validation.…”

“…Before undertaking any experimentation or sensitivity analysis with the model, it should be thoroughly tested (Tedeschi, 2006;Denil, Klikovits, Mosterman, Vallecillo, & Vangheluwe, 2017). Many researchers have devoted considerable attention to the process of model verification and validation (Tedeschi, 2006;McCoach & Black, 2008;Foures et al, 2016;Denil et al, 2017;Montgomery, 2017). However, despite all these studies, there still exists a requirement of a testing procedure which considers both the quantitative and qualitative methodology (Foures et al, 2016;Denil et al, 2017).…”

“…Many researchers have devoted considerable attention to the process of model verification and validation (Tedeschi, 2006;McCoach & Black, 2008;Foures et al, 2016;Denil et al, 2017;Montgomery, 2017). However, despite all these studies, there still exists a requirement of a testing procedure which considers both the quantitative and qualitative methodology (Foures et al, 2016;Denil et al, 2017). In this paper, the authors intend to develop a step by step procedure for model adequacy with the help of a practical case study on Model Adequacy for assessing the fit between the developed model using simulation with a well-established Mathematical Model (Markov model).…”