Optimization of Combinatorial Testing by Incremental SAT Solving

Yamada, Akihisa; Kitamura, Takashi; Artho, Cyrille; Choi, Eun-Hye; Oiwa, Yutaka; Biere, Armin

doi:10.1109/icst.2015.7102599

Cited by 45 publications

(41 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The test suites T 1 in Table III and T 2 in Table IV are 2-way (pairwise) test suites, since each of them covers all the possible parameter-value pairs in Table II. Many algorithms to efficiently construct small t-way test suites have been proposed so far. Approaches to generate tway test suites for SUT models with constraints include greedy algorithms (e. g., AETG [15], PICT [16] and ACTS [4]), heuristic search (e. g., CASA [19], HHSA [26], and TCA [32]), and SAT-based approaches (e. g., Calot [41]). …”

Section: A Combinatorial T-way Testingmentioning

confidence: 99%

See 1 more Smart Citation

Distance-Integrated Combinatorial Testing

Choi

Artho

Kitamura

et al. 2016

2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)

Self Cite

View full text Add to dashboard Cite

Abstract-This paper proposes a novel approach to combinatorial test generation, which achieves an increase of not only the number of new combinations but also the distance between test cases. We applied our distance-integrated approach to a stateof-the-art greedy algorithm for traditional combinatorial test generation by using two distance metrics, Hamming distance, and a modified chi-square distance. Experimental results using numerous benchmark models show that combinatorial test suites generated by our approach using both distance metrics can improve interaction coverage for higher interaction strengths with low computational overhead.

show abstract

Section: A Combinatorial T-way Testingmentioning

confidence: 99%

“…There have been a number of techniques and tools that generate t-way test suites, including greedy algorithms [14], [16], [31], heuristic search [19], [26], [39], and SAT-based approaches [24], [41]. These techniques, however, only ensure 100 % t-way coverage for given t, and do not try to improve t (> t)-way coverage.…”

Section: Related Workmentioning

confidence: 99%

Distance-Integrated Combinatorial Testing

Choi

Artho

Kitamura

et al. 2016

2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)

Self Cite

View full text Add to dashboard Cite

show abstract

“…CIT approaches to generate test cases can be divided in four main classes: Binary Decision Diagrams (BDDs) (Segall et al 2011), Satisfiability (SAT) solving (Cohen et al 1997;Yamada et al 2015;Yamada et al 2016), meta-heuristics (Garvin et al 2011;Shiba et al 2004;Hernandez et al 2010), and greedy algorithms (Lei and Tai 1998;Lei et al 2007) 1 . Recent CIT test case generation methods based on BDD and SAT are interesting to constrained (there are restrictions related to parameter interactions) problems but they perform worse compared with greedy algorithms/tools in the context of unconstrained (there are no restrictions at all) problems.…”

Section: Introductionmentioning

confidence: 99%

An algorithm for combinatorial interaction testing: definitions and rigorous evaluations

Balera

Júnior

2017

J Softw Eng Res Dev

View full text Add to dashboard Cite

Background: Combinatorial Interaction Testing (CIT) approaches have drawn attention of the software testing community to generate sets of smaller, efficient, and effective test cases where they have been successful in detecting faults due to the interaction of several input parameters. Recent empirical studies show that greedy algorithms are still competitive for CIT. It is thus interesting to investigate new approaches to address CIT test case generation via greedy solutions and to perform rigorous evaluations within the greedy context. Methods: We present a new greedy algorithm for unconstrained CIT, T-Tuple Reallocation (TTR), to generate CIT test suites specifically via the Mixed-value Covering Array (MCA) technique. The main reasoning behind TTR is to generate an MCA M by creating and reallocating t-tuples into this matrix M, considering a variable called goal (ζ ). We performed two controlled experiments addressing cost-efficiency and only cost. Considering both experiments, we did 3200 executions related to 8 solutions. In the first controlled experiment, we compared versions 1.1 and 1.2 of TTR in order to check whether there is significant difference between both versions of our algorithm. In such experiment, we jointly considered cost (size of test suites) and efficiency (time to generate the test suites) in a multi-objective perspective. In the second controlled experiment we confronted TTR 1.2 with five other greedy algorithms/tools for unconstrained CIT: IPOG-F, jenny, IPO-TConfig, PICT, and ACTS. We performed two different evaluations within this second experiment where in the first one we addressed cost-efficiency (multi-objective) and in the second only cost (single objective). Results: Results of the first controlled experiment indicate that TTR 1.2 is more adequate than TTR 1.1 especially for higher strengths (5, 6). In the second controlled experiment, TTR 1.2 also presents better performance for higher strengths (5, 6) where only in one case it is not superior (in the comparison with IPOG-F). We can explain this better performance of TTR 1.2 due to the fact that it no longer generates, at the beginning, the matrix of t-tuples but rather the algorithm works on a t-tuple by t-tuple creation and reallocation into M. Conclusion: Considering the metrics we defined in this work and based on both controlled experiments, TTR 1.2 is a better option if we need to consider higher strengths (5, 6). For lower strengths, other solutions, like IPOG-F, may be better alternatives.

show abstract

“…Indeed, CASA, TCA, and the SAT-based test suite optimization function in Calot [36] do not output a test suite even when they internally have one; they try to optimize it as far as possible, until certain termination conditions are met. Nevertheless, we observe some cases where our work may provide a benefit to these approaches; e. g., Calot previously employed ACTS for constructing an initial test suite for optimization, which is now done efficiently inside Calot.…”

Section: Constraints In Other Approachesmentioning

confidence: 99%

“…There is substantial work on combinatorial testing taking constraints and forbidden tuples into account, including meta-heuristic approaches [10,17,20,27], SAT-based approaches [29,36], and greedy approaches, which is further categorized into one-test-at-a-time (OTAT) approaches [9,11,10] and in-parameter-order generalized (IPOG) approaches [37,38].…”

Section: Mac Does Not Support Amd Cpusmentioning

confidence: 99%

Greedy combinatorial test case generation using unsatisfiable cores

Yamada

Biere

Artho

et al. 2016

Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

Self Cite

View full text Add to dashboard Cite

Combinatorial testing aims at covering the interactions of parameters in a system under test, while some combinations may be forbidden by given constraints (forbidden tuples).In this paper, we illustrate that such forbidden tuples correspond to unsatisfiable cores, a widely understood notion in the SAT solving community. Based on this observation, we propose a technique to detect forbidden tuples lazily during a greedy test case generation, which significantly reduces the number of required SAT solving calls. We further reduce the amount of time spent in SAT solving by essentially ignoring constraints while constructing each test case, but then "amending" it to obtain a test case that satisfies the constraints, again using unsatisfiable cores. Finally, to complement a disturbance due to ignoring constraints, we implement an efficient approximative SAT checking function in the SAT solver Lingeling.Through experiments we verify that our approach significantly improves the efficiency of constraint handling in our greedy combinatorial testing algorithm.

show abstract

Optimization of Combinatorial Testing by Incremental SAT Solving

Cited by 45 publications

References 25 publications

Distance-Integrated Combinatorial Testing

Distance-Integrated Combinatorial Testing

An algorithm for combinatorial interaction testing: definitions and rigorous evaluations

Greedy combinatorial test case generation using unsatisfiable cores

Contact Info

Product

Resources

About