Expanding our horizons in verification, validation, and accreditation research and practice

Balcı, Osman; Nance, Richard E.; Arthur, James D.; Ormsby, William F.

doi:10.1109/wsc.2002.1172944

Cited by 47 publications

(32 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, environmental sciences industry stakeholders have their own category but also frequently fund research activities (e.g., carbon trade companies invest in projects that reduce greenhouse gas emissions). The implication of stakeholders of different nature in this process would expand the horizon of model evaluation (after Balci and Ormsby 2002) up to considering aspects related to the specific context of research (Sect. 2.1), the credibility of model outputs when exploited for given purposes (Sect.…”

Section: Deliberative Processes For Comprehensive Model Evaluationmentioning

confidence: 99%

Deliberative processes for comprehensive evaluation of agroecological models. A review

Bellocchi

Rivington

Matthews

et al. 2014

Agron. Sustain. Dev.

View full text Add to dashboard Cite

The use of biophysical models in agroecology has increased in the last few decades for two main reasons: the need to formalize empirical knowledge and the need to disseminate model-based decision support for decision makers (such as farmers, advisors, and policy makers). The first has encouraged the development and use of mathematical models to enhance the efficiency of field research through extrapolation beyond the limits of site, season, and management. The second reflects the increasing need (by scientists, managers, and the public) for simulation experimentation to explore options and consequences, for example, future resource use efficiency (i.e., management in sustainable intensification), impacts of and adaptation to climate change, understanding market and policy responses to shocks initiated at a biophysical level under increasing demand, and limited supply capacity. Production concerns thus dominate most model applications, but there is a notable growing emphasis on environmental, economic, and policy dimensions. Identifying effective methods of assessing model quality and performance has become a challenging but vital imperative, considering the variety of factors influencing model outputs. Understanding the requirements of stakeholders, in respect of model use, logically implies the need for their inclusion in model evaluation methods. We reviewed the use of metrics of model evaluation, with a particular emphasis on the involvement of stakeholders to expand horizons beyond conventional structured, numeric analyses. Two major topics are discussed: (1) the importance of deliberative processes for model evaluation, and (2) the role computer-aided techniques may play to integrate deliberative processes into the evaluation of agroecological models. We point out that (i) the evaluation of agroecological models can be improved through stakeholder follow-up, which is a key for the acceptability of model realizations in practice, (ii) model credibility depends not only on the outcomes of well-structured, numerically based evaluation, but also on less tangible factors that may need to be addressed using complementary deliberative processes, (iii) comprehensive evaluation of simulation models can be achieved by integrating the expectations of stakeholders via a weighting system of preferences and perception, (iv) questionnaire-based surveys can help understand the challenges posed by the deliberative process, and (v) a benefit can be obtained if model evaluation is conceived in a decisional perspective and evaluation techniques are developed at the same pace with which the models themselves are created and improved. Scientific knowledge hubs are also recognized as critical pillars to advance good modeling practice in relation to model evaluation (including access to dedicated software tools), an activity which is frequently neglected in the context of time-limited framework programs.

show abstract

Section: Deliberative Processes For Comprehensive Model Evaluationmentioning

confidence: 99%

Deliberative processes for comprehensive evaluation of agroecological models. A review

Bellocchi

Rivington

Matthews

et al. 2014

Agron. Sustain. Dev.

View full text Add to dashboard Cite

show abstract

“…In order to execute this verification step before the runtime of the collaborative process, it is first carried out on a collaborative process model. Several verification techniques exist in the literature, including both informal techniques (testing, simulation and expertise) (Balci and Ornwsby 2002;Zeigler, Kim, and Praehofer 2000;Department of Defense 2003) and formal techniques (model checking, theorem proving) (Clarke, Grumbereg, and Doron 1999;Bérard et al 2001;Alur, Courcoubetis, and Dill 1993;Duffy 1991). Informal methods do not necessarily require the use of a model, but instead can be applied directly on the real system to establish the verification.…”

Section: Context and Problem Statementmentioning

confidence: 99%

Enabling model checking for collaborative process analysis: from BPMN to ‘Network of Timed Automata’

Mallek

Daclin

Chapurlat

et al. 2014

Enterprise Information Systems

View full text Add to dashboard Cite

Interoperability is a prerequisite for partners involved in performing collaboration. As a consequence, the lack of interoperability is now considered a major obstacle. The research work presented in this paper aims to develop an approach that allows specifying and verifying a set of interoperability requirements to be satisfied by each partner in the collaborative process prior to process implementation. To enable the verification of these interoperability requirements, it is necessary first and foremost to generate a model of the targeted collaborative process; for this research effort, the standardised language BPMN 2.0 is used. Afterwards, a verification technique must be introduced, and model checking is the preferred option herein. This paper focuses on application of the model checker UPPAAL in order to verify interoperability requirements for the given collaborative process model. At first, this step entails translating the collaborative process model from BPMN into a UPPAAL modelling language called 'Network of Timed Automata'. Second, it becomes necessary to formalise interoperability requirements into properties with the dedicated UPPAAL language, i.e. the temporal logic TCTL.

show abstract

“…The management and mitigation of risk was a focus of the Defense Modeling and Simulation Office's VV&A program, as described in [Youngblood, 2000]. Independent verification and validation (i.e., performed by an agent independent of the developer) is advocated as a means to reduce risk in [Balci, 2002].…”

Section: Selected Additional Relevant Literaturementioning

confidence: 99%

Defining the Meaning of a Major Modeling and Simulation Change as Applied to Accreditation

Petty

Alldredge

Beach³

et al. 2012

View full text Add to dashboard Cite

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. These existing methods vary widely in several respects including level of detail, specificity with respect to modeling and simulation, degree of quantitativeness, ease of use, and applicability to the sponsoring agency. All include some form of the notion of risk, which is conventionally defined as the product of the likelihood of an incorrect decision and the consequences of such a decision. A new method, the Quantitative-to-Qualitative Risk-based (QQR) method, was developed to make a quantitative recommendation regarding the re-validation of a modified model. The QQR method was developed with these goals in mind: to be quantitative, repeatable, and transparent to consider both model modifications and model use risk; to focus on model types and simulation applications of interest to the sponsoring agency; and to be simple and accessible so as to encourage its use in practical applications. The QQR method estimates the probability that not re-validating a modified model will lead to unacceptable consequences, given the modifications made to it. That estimated probability is meant to be interpreted, in the context of a re-validation decision, as a quantitative recommendation to re-validate the model. The method is based on a conditional probability formula that separates the various parts of the estimated probability into distinct terms and factors, and it provides procedures for estimating each term and factor. A central feature of the QQR method is a missions-means decomposition that allows both the method?s user to precisely and effectively identify the nature and extent of the modifications that were made to the model and the method to consider those modifications in its estimate. The QQR method was validated using a set of re-validation scenarios, each describing a model the modifications made to it, and the decision to be based on the model. The QQR method?s revalidation recommendations for the scenarios were compared to those of a set of human experts NO WARRANTY THIS STEVENS INSTITUTE OF TECHNOLOGY AND SYSTEMS ENGINEERING RESEARCH CENTER MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. STEVENS INSTITUTE OF TECHNOLOGY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDIN...

show abstract

Expanding our horizons in verification, validation, and accreditation research and practice

Cited by 47 publications

References 29 publications

Deliberative processes for comprehensive evaluation of agroecological models. A review

Deliberative processes for comprehensive evaluation of agroecological models. A review

Enabling model checking for collaborative process analysis: from BPMN to ‘Network of Timed Automata’

Defining the Meaning of a Major Modeling and Simulation Change as Applied to Accreditation

Contact Info

Product

Resources

About