Are We Prepared for Simulation Based Studies in Software Engineering Yet?

França, Breno Bernard Nicolau de; Travassos, Guilherme Horta

doi:10.19153/cleiej.16.1.8

Cited by 7 publications

(8 citation statements)

References 124 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the direction of these potential benefits, Software Engineering (SE) community has also presented some initiatives in using SBS to support investigation in the field. Indeed, apart from some interesting results, the SBS presented in the context of SE [3] allowed us to observe its initial maturity stage when compared with SBS concerned with the aforementioned areas. Lack of research protocols, ad-hoc experimental designs and output analysis, missing relevant information in the reports are some examples of observed issues into this context.…”

Section: Introductionmentioning

confidence: 99%

“…Based on the findings of our previous review [3] and on existing Empirical Software Engineering (ESE) guidelines for other investigation strategies, such as case studies and experiments, and simulation guidelines from other research areas, we proposed a preliminary set of guidelines aiming at providing guidance to researchers when reporting SBS into the SE context [4]. Later, we performed a first assessment of this set of reporting guidelines based on the approach presented in [5].…”

Section: Introductionmentioning

confidence: 99%

“…In addition, it is also important to depict the differences between common threats to validity (as those usually observed at in vivo and in vitro studies), and highlight those ones specifically identified at in virtuo and in silico studies. Therefore, we have conducted a secondary analysis of the data collected in [3] under the perspective of potential threats to validity found in SBS, which we are now presenting in this paper. As far as we are aware, there is no other work like this into the context of Experimental Software Engineering involving SBS.…”

Section: Introductionmentioning

confidence: 99%

“…The work presented in this paper comprehends a broader effort in trying to organize a body of knowledge regarding SBS in the context of SE. Apart from earlier motivations and previous work on simulation models [6] [7], we undertook a systematic literature review aiming at characterizing how different simulation approaches have been used in SE studies [3] following the guidelines proposed by [8] and adopting the PICO [9] strategy to structure the search string. In this review, population, intervention and outcome dimensions are considered to support the answer of the research question.…”

Section: Introductionmentioning

confidence: 99%

“…For that, we primarily based our search in the outcomes from the quasi-systematic literature review. However, as previously observed in [3], the terminology is not consensual and authors in this field rarely discuss threats to validity using terms such as "threats to validity" or related ones. Thus, we decided to apply a systematic approach to handle the threats descriptions under the same perspective.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Simulation Based Studies in Software Engineering: A Matter of Validity

França

Travassos

2015

CLEIej

View full text Add to dashboard Cite

CONTEXT: Despite the possible lack of validity when compared with other science areas, Simulation-Based Studies (SBS) in Software Engineering (SE) have supported the achievement of some results in the field. However, as it happens with any other sort of experimental study, it is important to identify and deal with threats to validity aiming at increasing their strength and reinforcing results confidence. OBJECTIVE: To identify potential threats to SBS validity in SE and suggest ways to mitigate them. METHOD: To apply qualitative analysis in a dataset resulted from the aggregation of data from a quasi-systematic literature review combined with ad-hoc surveyed information regarding other science areas. RESULTS: The analysis of data extracted from 15 technical papers allowed the identification and classification of 28 different threats to validity concerned with SBS in SE according Cook and Campbell’s categories. Besides, 12 verification and validation procedures applicable to SBS were also analyzed and organized due to their ability to detect these threats to validity. These results were used to make available an improved set of guidelines regarding the planning and reporting of SBS in SE. CONCLUSIONS: Simulation based studies add different threats to validity when compared with traditional studies. They are not well observed and therefore, it is not easy to identify and mitigate all of them without explicit guidance, as the one depicted in this paper.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Simulation Based Studies in Software Engineering: A Matter of Validity

França

Travassos

2015

CLEIej

View full text Add to dashboard Cite

show abstract

Metrics for software process simulation modeling

Liu,

Zhang,

Dong

et al. 2024

J Software Evolu Process

View full text Add to dashboard Cite

Software process simulation (SPS) has become an effective tool for software process management and improvement. However, its adoption in industry is less than what the research community expected due to the burden of measurement cost and the high demand for domain knowledge. The difficulty of extracting appropriate metrics with real data from process enactment is one of the great challenges. We aim to provide evidence‐based support of the process metrics for software process (simulation) modeling. A systematic literature review was performed by extending our previous review series to draw a comprehensive understanding of the metrics for process modeling following our proposed ontology of metrics in SPS. We identify 131 process modeling studies that collectively involve 1975 raw metrics and classified them into 21 categories using the coding technique. We found product and process external metrics are not used frequently in SPS modeling while resource external metrics are widely used. We analyze the causal relationships between metrics. We find that the models exhibit significant diversity, as no pairwise relationship between metrics accounts for more than 10% SPS models. We identify 17 data issues may encounter in measurement and 10 coping strategies. The results of this study provide process modelers with an evidence‐based reference of the identification and the use of metrics in SPS modeling and further contribute to the development of the body of knowledge on software metrics in the context of process modeling. Furthermore, this study is not limited to process simulation but can be extended to software process modeling, in general. Taking simulation metrics as standards and references can further motivate and guide software developers to improve the collection, governance, and application of process data in practice.

show abstract

Experimentation with dynamic simulation models in software engineering: planning and reporting guidelines

França

Travassos

2015

Empir Software Eng

View full text Add to dashboard Cite

Simulation-based studies (SBS) have become an interesting investigation approach for Software Engineering (SE). However, the reports on experiments with dynamic simulation models found in the technical literature lack relevant information, hampering the full understanding of the procedures and results reported, as well as their replicability. Apart from the limitations on the length in conferences and journal papers, some of the relevant information seems to be missing due to methodological issues not considered when conducting such studies. This is the case of missing research questions and goals, lack of evidence regarding the dynamic simulation model validity, poorly designed simulation experiments, amongst others. Based on findings from a previous quasi-systematic literature review, we propose a set of reporting guidelines for SBS with dynamic models in the context of SE aiming at providing guidance on which information the report should contain. Furthermore, these guidelines were evolved to support SBS planning by identifying potential threats to simulation study validity and in making recommendations to avoid them, through qualitative analysis and external evaluation. Finally, we conducted different evaluations regarding both the reporting and planning guidelines, apart from using them to support the planning of a SBS as regards software evolution. A set of 33 reporting and planning guidelines for different stages of the simulation lifecycle and focused on the experimentation with dynamic simulation models have been put together. The first assessments point to a comprehensive set of guidelines, supporting a comprehensive preparation and review of the plans and reports from the studies, apart from the planning of a SBS focused on software evolution, potentially reducing the threats to the experimentation with the validity of dynamic simulation models. The 33 guidelines cannot be understood as separate groups for reporting and planning as they overlap in many aspects. The Empir Software Eng main goal is to use the guidelines to support the planning of a simulation-based study with dynamic models so that experimenters may identify potential threats to validity and produce relevant information for a complete simulation experiment report in advance. Despite their initial contribution to increase the validity of SBS, the reporting and planning of simulationbased experiments with dynamic models still has to be discussed and improved in SE. Therefore, additional assessments of this set of guidelines are needed to strengthen the confidence in their completeness and usefulness.

show abstract

Are We Prepared for Simulation Based Studies in Software Engineering Yet?

Cited by 7 publications

References 124 publications

Simulation Based Studies in Software Engineering: A Matter of Validity

Simulation Based Studies in Software Engineering: A Matter of Validity

Metrics for software process simulation modeling

Experimentation with dynamic simulation models in software engineering: planning and reporting guidelines

Contact Info

Product

Resources

About