Facilitating Parameter Estimation and Sensitivity Analysis of Agent-Based Models: A Cookbook Using NetLogo and 'R'

Abstract

Agent-based models are increasingly used to address questions regarding real-world phenomena and mechanisms; therefore, the calibration of model parameters to certain data sets and patterns is often needed. Furthermore, sensitivity analysis is an important part of the development and analysis of any simulation model. By exploring the sensitivity of model output to changes in parameters, we learn about the relative importance of the various mechanisms represented in the model and how robust the model output is to parameter uncertainty. These insights foster the understanding of models and their use for theory development and applications. Both steps of the model development cycle require massive repetitions of simulation runs with varying parameter values. To facilitate parameter estimation and sensitivity analysis for agent-based modellers, we show how to use a suite of important established methods. Because NetLogo and R are widely used in agent-based modelling and for statistical analyses, we use a simple model implemented in NetLogo as an example, packages in R that implement the respective methods, and the RNetLogo package, which links R and NetLogo. We briefly introduce each method and provide references for further reading. We then list the packages in R that may be used for implementing the methods, provide short code examples demonstrating how the methods can be applied in R, and present and discuss the corresponding outputs. The Supplementary Material includes full, adaptable code samples for using the presented methods with R and NetLogo. Our overall aim is to make agent-based modellers aware of existing methods and tools for parameter estimation and sensitivity analysis and to provide accessible tools for using these methods. In this way, we hope to contribute to establishing an advanced culture of relating agent-based models to data and patterns observed in real systems and to foster rigorous and structured analyses of agent-based models.

Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation

“…More research is needed to find an optimum strategy for TDSM that balances test coverage and cost. Techniques such as Latin Hypercube described in Thiele et al (2014) can be used to address this problem. Thirdly, a unit test can contain mistakes too.…”

“…This comparison is closely linked to black-box testing. It should be noted that model validation often includes sensitivity analysis that explores the behaviour of the model under various combinations of parameter values (within acceptable ranges) as discussed in Thiele et al (2014). In this case, a validation case tests the behaviour of the model under a specific combination of parameter values.…”

“…Hence, the number of validation cases grows exponentially with the number of parameters. For this reason, in addition to better exploration algorithms, Thiele et al (2014) highlight the importance of an automated tool (such as our tool) for model validation. As shown in Figure 5, we start with the specification of validation cases in close collaboration with model users (and/or domain experts).…”

“…Modellers are forced to think about how their model is going to be verified and validated, even before they begin to develop it. The second advantage is that TDSM can be implemented using various unit-testing tools, or incorporated into a framework such as the one proposed by Gurcan et al (2013), Niazzi et al (2009) and Thiele et al (2014). To demonstrate how the proposed validation technique can be used in practice, we will show a few examples in which parts of the agent-based simulation software called MASSIM (MAritime Search SIMulation) are verified and validated using TDSM.…”

EvoPER -- An R package for applying evolutionary computation methods in the parameter estimation of individual-based models implemented in Repast

“…In other words, the 72 metric gives a numerical hint about how close are the output of model to the reference data. There are 73 fundamentally three approaches to define the goodness of fit for a model (Thiele et al (2014)). The first 74 approach is based on using acceptable ranges for the model outputs being the most straightforward one.…”