Statistical analysis of simulation output: State of the art

Alexopoulos,

doi:10.1109/wsc.2007.4419597

Cited by 13 publications

(8 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, we searched for simulation studies in the main conference (ICSSP 3 ) and journal (SPIP, 4 including its new title JSEP), applying the same criteria from the qSLR. We found seven other simulation studies(Al-Emran et al 2010;Birkhölzer et al 2010;Houston and Lieu 2010;Bai et al 2012;Paikari et al 2012;Concas et al 2013;Houston and Buettner 2013) in these venues.…”

mentioning

confidence: 88%

See 1 more Smart Citation

Experimentation with dynamic simulation models in software engineering: planning and reporting guidelines

França

Travassos

2015

Empir Software Eng

View full text Add to dashboard Cite

Simulation-based studies (SBS) have become an interesting investigation approach for Software Engineering (SE). However, the reports on experiments with dynamic simulation models found in the technical literature lack relevant information, hampering the full understanding of the procedures and results reported, as well as their replicability. Apart from the limitations on the length in conferences and journal papers, some of the relevant information seems to be missing due to methodological issues not considered when conducting such studies. This is the case of missing research questions and goals, lack of evidence regarding the dynamic simulation model validity, poorly designed simulation experiments, amongst others. Based on findings from a previous quasi-systematic literature review, we propose a set of reporting guidelines for SBS with dynamic models in the context of SE aiming at providing guidance on which information the report should contain. Furthermore, these guidelines were evolved to support SBS planning by identifying potential threats to simulation study validity and in making recommendations to avoid them, through qualitative analysis and external evaluation. Finally, we conducted different evaluations regarding both the reporting and planning guidelines, apart from using them to support the planning of a SBS as regards software evolution. A set of 33 reporting and planning guidelines for different stages of the simulation lifecycle and focused on the experimentation with dynamic simulation models have been put together. The first assessments point to a comprehensive set of guidelines, supporting a comprehensive preparation and review of the plans and reports from the studies, apart from the planning of a SBS focused on software evolution, potentially reducing the threats to the experimentation with the validity of dynamic simulation models. The 33 guidelines cannot be understood as separate groups for reporting and planning as they overlap in many aspects. The Empir Software Eng main goal is to use the guidelines to support the planning of a simulation-based study with dynamic models so that experimenters may identify potential threats to validity and produce relevant information for a complete simulation experiment report in advance. Despite their initial contribution to increase the validity of SBS, the reporting and planning of simulationbased experiments with dynamic models still has to be discussed and improved in SE. Therefore, additional assessments of this set of guidelines are needed to strengthen the confidence in their completeness and usefulness.

show abstract

mentioning

confidence: 88%

“…Authors usually refer to them as limitations and unverified assumptions, without discussing their consequences. Al-Emran et al (2010) and Concas et al (2013) present discussions according to the types of threats proposed in (Cook and Campbell 1979).…”

Section: Analysis Against the Technical Literaturementioning

confidence: 99%

Experimentation with dynamic simulation models in software engineering: planning and reporting guidelines

França

Travassos

2015

Empir Software Eng

View full text Add to dashboard Cite

show abstract

“…For example, where ( Y m ) n is the output data at simulation replication n , and N is the number of replications. Further, we can compute a confidence interval for the expectation; see, e.g., [ 43 ] for more details about transient simulation analysis. A crucial point here is to determine the required amount of simulation replications (or samples) in order to achieve a reliable estimation of the measure of interest.…”

Section: Simulation and Analysismentioning

confidence: 99%

Fuzzy Stochastic Petri Nets for Modeling Biological Systems with Uncertain Kinetic Parameters

2016

View full text Add to dashboard Cite

Stochastic Petri nets (SPNs) have been widely used to model randomness which is an inherent feature of biological systems. However, for many biological systems, some kinetic parameters may be uncertain due to incomplete, vague or missing kinetic data (often called fuzzy uncertainty), or naturally vary, e.g., between different individuals, experimental conditions, etc. (often called variability), which has prevented a wider application of SPNs that require accurate parameters. Considering the strength of fuzzy sets to deal with uncertain information, we apply a specific type of stochastic Petri nets, fuzzy stochastic Petri nets (FSPNs), to model and analyze biological systems with uncertain kinetic parameters. FSPNs combine SPNs and fuzzy sets, thereby taking into account both randomness and fuzziness of biological systems. For a biological system, SPNs model the randomness, while fuzzy sets model kinetic parameters with fuzzy uncertainty or variability by associating each parameter with a fuzzy number instead of a crisp real value. We introduce a simulation-based analysis method for FSPNs to explore the uncertainties of outputs resulting from the uncertainties associated with input parameters, which works equally well for bounded and unbounded models. We illustrate our approach using a yeast polarization model having an infinite state space, which shows the appropriateness of FSPNs in combination with simulation-based analysis for modeling and analyzing biological systems with uncertain information.

show abstract

“…First in Section 2 we review the estimators that will be used in the paper (for recent surveys about of the general state-of-the-art in the areas, see e.g. Alexopoulos, 2007, andLaw, 2007). Then in Section 3 we give some numerical examples, in which the method of batch means performs better than regenerative simulation.…”

Section: To Cite This Versionmentioning

confidence: 99%

“…Alexopoulos (2007) describes elaborate algorithms for adaptively choosing the number of batches; our aim here is however not to use the most sophisticated algorithms, but rather to show that even with a simple choice for the number of batches, this approach performs better than intervals based on regenerative simulation.…”

Section: Numerical Examplesmentioning

confidence: 99%