Experimentation with dynamic simulation models in software engineering: planning and reporting guidelines

Abstract

Simulation-based studies (SBS) have become an interesting investigation approach for Software Engineering (SE). However, the reports on experiments with dynamic simulation models found in the technical literature lack relevant information, hampering the full understanding of the procedures and results reported, as well as their replicability. Apart from the limitations on the length in conferences and journal papers, some of the relevant information seems to be missing due to methodological issues not considered when conducting such studies. This is the case of missing research questions and goals, lack of evidence regarding the dynamic simulation model validity, poorly designed simulation experiments, amongst others. Based on findings from a previous quasi-systematic literature review, we propose a set of reporting guidelines for SBS with dynamic models in the context of SE aiming at providing guidance on which information the report should contain. Furthermore, these guidelines were evolved to support SBS planning by identifying potential threats to simulation study validity and in making recommendations to avoid them, through qualitative analysis and external evaluation. Finally, we conducted different evaluations regarding both the reporting and planning guidelines, apart from using them to support the planning of a SBS as regards software evolution. A set of 33 reporting and planning guidelines for different stages of the simulation lifecycle and focused on the experimentation with dynamic simulation models have been put together. The first assessments point to a comprehensive set of guidelines, supporting a comprehensive preparation and review of the plans and reports from the studies, apart from the planning of a SBS focused on software evolution, potentially reducing the threats to the experimentation with the validity of dynamic simulation models. The 33 guidelines cannot be understood as separate groups for reporting and planning as they overlap in many aspects. The Empir Software Eng main goal is to use the guidelines to support the planning of a simulation-based study with dynamic models so that experimenters may identify potential threats to validity and produce relevant information for a complete simulation experiment report in advance. Despite their initial contribution to increase the validity of SBS, the reporting and planning of simulationbased experiments with dynamic models still has to be discussed and improved in SE. Therefore, additional assessments of this set of guidelines are needed to strengthen the confidence in their completeness and usefulness.

Modeling and Simulation of a Smart Street Lighting System

“…The in-silico experiment was conducted according to França and Travassos guidelines [de França and Travassos 2016]. Our study consists of simulating part of an Smart Public Lighting System (PSLS), responsible for changing the behavior of the luminaires located in the vicinity of traffic accidents, thus providing an alert zone for drivers traveling through this region, avoiding further accidents.…”

Stimuli-SoS: a model-based approach to derive stimuli generators for simulations of systems-of-systems software architectures

“…Recent studies have investigated the adoption of simulation in software engineering [63], and simulation has certainly been applied for SoS development [16,36,64]. Additionally, initiatives have invested in the simulation of software architectures, but not specifically for software architectures of SoS, such as SySML [65], MatLab/Simulink [66], Palladio 22 [67], Bogado et al [36], and Alexander et al [68].…”

“…Simulations have been recognized as source of empirical evidences for software engineering [63]. Hence, the adoption of our approach can leverage the research on empirical software engineering supported by simulations.…”

ASAS: An Approach to Support Simulation of Smart Systems

“…However, they lack mechanisms to capture SoS dynamics, hampering evaluation of the operation of SoS [8]. Simulations provide the means to evaluate SoS operation by anticipating failures and unveiling dynamic aspects of systems' operation [5,13,17]. Therefore, we pose the following research question: Is it possible to automatically generate simulation models for a SoS to support evaluating its operation?…”

“…However, their proposal mentions drawbacks related to: (i) use of an informal description of the SoS architecture with a notation based on UML, and (ii) lack of dynamic and functional properties. Xia et al [19] evaluated the SoS architectures documented in DoDAF 5 by means of a model-driven approach that transforms system architecture models in Simulink 6 into executable models. Their approach focuses on measuring non-functional properties, such as feasibility and efficiency, but it does not consider software architectures.…”