Valid and Reproducible Simulation Studies—Making It Explicit

Reinhardt, Oliver; Warnke, Tom; Ruscheinski, Andreas; Uhrmacher, Adelinde M.

doi:10.1007/978-3-319-70766-2_25

“…Validation is an important task in the modeling and simulation life cycle. It is the process of substantiating whether the model behaves consistently with our expectations, e.g., regarding data that has been measured in the real system [54].…”

Section: Repeated Model Validation After Model Extension and Compositionmentioning

confidence: 66%

“…Calibration, also called model fitting, is the process of finding a parameterization of the model which can reproduce observed behavior of the real system [54]. While calibration and validation both typically relate to real data, the conclusions drawn from them are essentially different.…”

Section: Repeated Model Calibrationmentioning

confidence: 99%

Automatic Reuse, Adaption, and Execution of Simulation Experiments via Provenance Patterns

Wilsdorf

¹

,

Wolpers

²

,

Hilton

³

et al. 2023

ACM Trans. Model. Comput. Simul.

8

0

View full text Add to dashboard Cite

Simulation experiments are typically conducted repeatedly during the model development process, for example, to re-validate if a behavioral property still holds after several model changes. Approaches for automatically reusing and generating simulation experiments can support modelers in conducting simulation studies in a more systematic and effective manner. They rely on explicit experiment specifications and, so far, on user interaction for initiating the reuse. Thereby, they are constrained to support the reuse of simulation experiments in a specific setting. Our approach now goes one step further by automatically identifying and adapting the experiments to be reused for a variety of scenarios. To achieve this, we exploit provenance graphs of simulation studies, which provide valuable information about the previous modeling and experimenting activities, and contain meta-information about the different entities that were used or produced during the simulation study. We define provenance patterns and associate them with a semantics, which allows us to interpret the different activities, and construct transformation rules for provenance graphs. Our approach is implemented in a Reuse and Adapt framework for Simulation Experiments (RASE) which can interface with various modeling and simulation tools. In the case studies, we demonstrate the utility of our framework for a) the repeated sensitivity analysis of an agent-based model of migration routes, and b) the cross-validation of two models of a cell signaling pathway.

show abstract

“…For documenting the simulation model and its foundation in economic theory we complement a textual description of the model with a provenance model [ 32 , 33 ]. While the former focuses on the mechanics of the model, the latter captures information about the context of the model, and the process of its creation.…”

Section: Methodsmentioning

confidence: 99%

A basic macroeconomic agent-based model for analyzing monetary regime shifts

Peters

¹

,

Neuberger

²

,

Reinhardt

³

et al. 2022

PLoS ONE

1

0

View full text Add to dashboard Cite

In macroeconomics, an emerging discussion of alternative monetary systems addresses the dimensions of systemic risk in advanced financial systems. Monetary regime changes with the aim of achieving a more sustainable financial system have already been discussed in several European parliaments and were the subject of a referendum in Switzerland. However, their effectiveness and efficacy concerning macro-financial stability are not well-known. This paper defines the economic requirements for modeling the current monetary system and introduces the corresponding macroeconomic agent-based model (MABM) in a continuous-time stochastic agent-based simulation environment with a provenance model. This MABM aims to present a starting point for exploring and analyzing monetary reforms. In this context, the monetary system affects the lending potential of banks and might impact the dynamics of financial crises. MABMs are predestined to replicate emergent financial crisis dynamics, analyze institutional changes within a financial system, and thus measure macro-financial stability. The used simulation environment makes the model more accessible and facilitates exploring the impact of different hypotheses and mechanisms in a less complex way. Moreover, the model replicates a wide range of stylized economic facts, which validates it as an analysis tool to implement and compare monetary regime shifts.

show abstract

“…Validation is an important task in the modeling and simulation life cycle. It is the process of substantiating whether the model behaves consistently with our expectations, e.g., regarding data which has been measured in the real system Reinhardt et al [2019a]. However, the model development is usually not completed after the first validation.…”

Section: Repeated Model Validation (Regression Testing)mentioning

confidence: 97%

“…Calibration, also called model fitting, is the process of finding a parameterization of the model which can reproduce observed behavior of the real system Reinhardt et al [2019a]. While calibration and validation both typically relate to real data, the conclusion drawn from them are essentially different.…”

Section: Repeated Model Calibrationmentioning

confidence: 99%

Automatic Reuse, Adaption, and Execution of Simulation Experiments via Provenance Patterns

Wilsdorf¹,

Wolpers²,

Hilton³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Simulation experiments are typically conducted repeatedly during the model development process, for example, to re-validate if a behavioral property still holds after several model changes. Approaches for automatically reusing and generating simulation experiments can support modelers in conducting simulation studies in a more systematic and effective manner. They rely on explicit experiment specifications and, so far, on user interaction for initiating the reuse. Thereby, they are constrained to support the reuse of simulation experiments in a specific setting. Our approach now goes one step further by automatically identifying and adapting the experiments to be reused for a variety of scenarios. To achieve this, we exploit provenance graphs of simulation studies, which provide valuable information about the previous modeling and experimenting activities, and contain meta-information about the different entities that were used or produced during the simulation study. We define provenance patterns and associate them with a semantics, which allows us to interpret the different activities, and construct transformation rules for provenance graphs. Our approach is implemented in a Reuse and Adapt framework for Simulation Experiments (RASE) which can interface with various modeling and simulation tools. In the case studies, we demonstrate the utility of our framework for a) the repeated sensitivity analysis of an agent-based model of migration routes, and b) the cross-validation of two models of a cell signaling pathway.

show abstract

Valid and Reproducible Simulation Studies—Making It Explicit

Cited by 3 publications

References 37 publications

Automatic Reuse, Adaption, and Execution of Simulation Experiments via Provenance Patterns

Automatic Reuse, Adaption, and Execution of Simulation Experiments via Provenance Patterns

A basic macroeconomic agent-based model for analyzing monetary regime shifts

Automatic Reuse, Adaption, and Execution of Simulation Experiments via Provenance Patterns

Contact Info

Product

Resources

About