How to Build Valid and Credible Simulation Models

Abstract

In this tutorial we present techniques for building valid and credible simulation models. Ideas to be discussed include the importance of a definitive problem formulation, discussions with subject-matter experts, interacting with the decision-maker on a regular basis, development of a written conceptual model, structured walk-through of the conceptual model, use of sensitivity analysis to determine important model factors, and comparison of model and system output data for an existing system (if any). Each idea will be illustrated by one or more real-world examples. We will also discuss the difficulty in using formal statistical techniques (e.g., confidence intervals) to validate simulation models.

An agent‐based simulation model of Eldana saccharina Walker

“…The statecharts represent the combination of three of the building blocks discussed previously (the life cycle, the mating process and the oviposition and dispersion process) and agents reside in and are affected by the environment constructed within the GUI. The operation of the model was verified, both incrementally during construction, as well as in the case of a complete system, using typical techniques described in the literature (Law, 1990;Sargent, 2005).…”

Sensitivity analysis of Repast computational ecology models with R/Repast

“…The two main objectives of sensitivity analysis are understanding how robust are the model results considering the existing uncertainties and quantifying the effect of input factors on the variance of output (Law, 2005;Pianosi et al, 2016;Saltelli et al, 2004). The intrinsic characteristics of individual-based models which relies on mechanistic descriptions favors the production of models with many subprocesses, state variable, and parameters.…”

A review of performance criteria to validate simulation models

“…α = 0.05). As stressed by Law (2005), caution is needed on the direct application of classical statistical tests to validate simulation models, as these tests are based on independent, identically distributed observations, and the outputs under analysis (i.e. observed and simulated data sets) are often nonstationary (i.e.…”

“…the distributions of the successive observations change over time) and auto-correlated (the observations in the process are correlated with each other). Law (2005) stresses that the null hypotheses of equal outputs from reality and simulation are necessarily false, because a simulation model is only an approximation of reality, suggesting that it is better to assess if the differences between the model and reality are significant enough to affect any conclusions derived from the model. The statistical tests reviewed to assess bias are the sign test (for similarity of medians), the Student's t-test and the Wilcoxon signed-rank test (for similarity of means).…”

“…The literature on simulation models includes distinct meanings for the term validation, which is normally set alongside with the concepts of quality (Sterling & Taveter, 2009), verification, confirmation and calibration (Oreskes et al, 1994;Refsgaard & Henriksen, 2004). A simulation model is a simplified yet valid reproduction of a real system under study (Law, 2005). The exhaustive examination of a simulation model is not possible, recognizing that any theory could ever be exposed to all possible and relevant tests (Kuhn, 1962).…”