Managing Conflicts When Using Combination Strategies to Test Software

Abstract

Testers often represent systems under test in input parameter models. These contain parameters with associated values. Combinations of parameter values, with one value for each parameter, are potential test cases. In most models, some values of two or more parameters cannot be combined. Testers must then detect and avoid or remove these conflicts.This paper proposes two new methods for automatically handling such conflicts and compares these with two existing methods, based on the sizes of the final conflict-free test suites. A test suite reduction method, usable with three of the four investigated methods is also included in the study, resulting in seven studied conflict handling methods.In the experiment, the number and types of conflicts, as well as the size of the input parameter model and the coverage criterion used, are varied. All in all, 3854 test suites with a total of 929, 158 test cases were generated.Two methods stand out as tractable and complementary. The best method (called the avoid methods) with respect to test suite size is to avoid selection of test cases with conflicts. However, this method cannot always be used. The second best method (called the replace method), removing conflicts from the final test suite, is completely general.

Dealing with Constraints in Boolean Expression Testing

“…A similar conclusion is drawn by Grindal et alt. [12]: they compare four strategies to handle conflicts in combinatorial testing (using also a post reduction technique similar to that presented in this paper) and conclude that "the best method with respect to test suite size is to avoid selection of test cases with conflicts". They consider only test suite sizes and they do not, however, consider the fault detection capability when comparing the strategies.…”

Design and implementation of a t-way test data generation strategy with automated execution tool support

“…On a positive note, computational approaches can be applied to arbitrary system configurations. Furthermore, computational approaches are more adaptable for constraint handling [18,25] and test prioritization [4].…”

Handling Conflicts to Test Transport Protocol's Parallel Routing on a Vehicle Gateway System

“…The maximum number of resulting combinations (rows) in this example can be calculated according to (6) and is equal to 20. While generating test cases, following aspects are considered:…”

“…A conflict in an IPM is due to an invalid combination of input parameter values and hence this combination must be omitted or avoided while generating test cases. Diverse conflict handling strategies such as sub-models and avoid have been suggested in literature [6] to overcome conflicts in IPMs.…”