Ensuring the Observability of Structural Test Obligations

Meng, Ying; Whalen, Michael W.

doi:10.1109/tse.2018.2869146

Cited by 8 publications

(10 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this study, we make use of 37 models from the open-source Lustre Benchmarks dataset 3 . This dataset has been used in previous test generation experiments [38], and includes complex models such as Docking_Approach, a NASA-created example that describes the behavior of a space shuttle as it docks with the International Space Station [24]. Another model, Infusion_Manager represents 1 http://www.mathworks.com/products/simulink 2 http://www.mathworks.com/stateflow 3 Available from https://github.com/Greg4cr/Reworked-Benchmarks/tree/SingleNode.…”

Section: Case Examplesmentioning

confidence: 99%

“…We use a counterexample-based test generation framework for Lustre programs used in past research on test generation 4 [21,24,38]. This framework generates counterexamples using the JKind bounded model checker [19,26], then fills in default values for any variables not specified by the specification.…”

Section: Test Input Generationmentioning

confidence: 99%

“…The solver is invoked for each goal, meaning that this form of test generation will produce one test per testing goal. There may be overlap between tests in terms of goals covered, and test suite reduction is often used to narrow the size of the suite [38]. As suite reduction adds stochastic noise to experiment results, we do not employ it in this study, and instead examine the full suite.…”

Section: Test Input Generationmentioning

confidence: 99%

“…This does not mean these faults are unimportant. Rather, it means that the chosen input was not sufficient to reveal the fault [21,24,38].…”

Section: Choice Of Test Oraclementioning

confidence: 99%

See 3 more Smart Citations

Understanding The Impact of Solver Choice in Model-Based Test Generation

Meng

2020

Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

Self Cite

View full text Add to dashboard Cite

Background: In model-based test generation, SMT solvers explore the state-space of the model in search of violations of specified properties. If the solver finds that a predicate can be violated, it produces a partial test specification demonstrating the violation. Aims: The choice of solvers is important, as each may produce differing counterexamples. We aim to understand how solver choice impacts the effectiveness of generated test suites at finding faults. Method: We have performed experiments examining the impact of solver choice across multiple dimensions, examining the ability to attain goal satisfaction and fault detection when satisfaction is achieved-varying the source of test goals, data types of model input, and test oracle. Results: The results of our experiment show that solvers vary in their ability to produce counterexamples, and-for models where all solvers achieve goal satisfaction-in the resulting fault detection of the generated test suites. The choice of solver has an impact on the resulting test suite, regardless of the oracle, model structure, or source of testing goals. Conclusions: The results of this study identify factors that impact fault-detection effectiveness, and advice that could improve future approaches to model-based test generation. CCS CONCEPTS• Software and its engineering → Software testing and debugging; System modeling languages.

show abstract

Section: Case Examplesmentioning

confidence: 99%

Section: Test Input Generationmentioning

confidence: 99%

Section: Test Input Generationmentioning

confidence: 99%

“…This does not mean these faults are unimportant. Rather, it means that the chosen input was not sufficient to reveal the fault [21,24,38].…”

Section: Choice Of Test Oraclementioning

confidence: 99%

See 2 more Smart Citations

Understanding The Impact of Solver Choice in Model-Based Test Generation

Meng

2020

Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The output-only oracle that specifies output variables as oracle data is the most common type of test oracle [5]. Nevertheless, in some cases, faults leading to incorrect states do not always propagate and manifest themselves as failures and may vanish before they reach output variables [6]. Except for the propagation problems of output-only oracles, it is difficult to determine the expected values of the output of programs under test because of the complexity of programs [7].…”

Section: Introductionmentioning

confidence: 99%

Path-Sensitive Oracle Data Selection via Static Analysis

et al. 2021

View full text Add to dashboard Cite

A test oracle is a procedure that is used during testing to determine whether software behaves correctly or not. One of most important tasks for a test oracle is to choose oracle data (the set of variables monitored during testing) to observe. However, most literature on test oracles has focused either on formal specification generation or on automated test oracle construction, whereas little work exists for supporting oracle data selection. In this paper, we present a path-sensitive approach, PSODS (path-sensitive oracle data selection), to automatically select oracle data for use by expected value oracles. PSODS ranks paths according to the possibility that potential faults may exist in them, and the ranked paths help testers determine which oracle data should be considered first. To select oracle data for each path, we introduce quantity and quality analysis of oracle data, which use static analysis to estimate oracle data for their substitution capability and fault-detection capability. Quantity analysis can reduce the number of oracle data. Quality analysis can rank oracle data based on their fault-detection capability. By using quantity and quality analysis, PSODS reduces the cost of oracle construction and improves fault-detection efficiency and effectiveness. We have implemented our approach and applied it to a real-world project. The experimental results show that PSODS is efficient in helping testers construct test oracles. Moreover, the oracle datasets produced by our approach are more effective and efficient than output-only oracles at detecting faults.

show abstract

Generation Method of Nuclear Power DCS Integrated Test Case Based on Symbolic Execution

Liu

et al. 2022

Lecture Notes in Electrical Engineering

View full text Add to dashboard Cite

Ensuring the Observability of Structural Test Obligations

Cited by 8 publications

References 64 publications

Understanding The Impact of Solver Choice in Model-Based Test Generation

Understanding The Impact of Solver Choice in Model-Based Test Generation

Path-Sensitive Oracle Data Selection via Static Analysis

Generation Method of Nuclear Power DCS Integrated Test Case Based on Symbolic Execution

Contact Info

Product

Resources

About