Generation and Application of Constrained Interaction Test Suites Using Base Forbidden Tuples with a Mixed Neighborhood Tabu Search

Abstract

To ensure the quality of current highly configurable software systems, intensive testing is needed to test all the configuration combinations and detect all the possible faults. This task becomes more challenging for most modern software systems when constraints are given for the configurations. Here, intensive testing is almost impossible, especially considering the additional computation required to resolve the constraints during the test generation process. In addition, this testing process is exhaustive and time-consuming. Combinatorial interaction strategies can systematically reduce the number of test cases to construct a minimal test suite without affecting the effectiveness of the tests. This paper presents a new efficient searchbased strategy to generate constrained interaction test suites to cover all possible combinations. The paper also shows a new application of constrained interaction testing in software fault searches. The proposed strategy initially generates the set of all possible t − tuple combinations; then, it filters out the set by removing the forbidden t − tuples using the base forbidden tuple (BFT) approach. The strategy also utilizes a mixed neighborhood tabu search (TS) to construct optimal or near-optimal constrained test suites. The efficiency of the proposed method is evaluated through a comparison against two well-known state-of-the-art tools. The evaluation consists of three sets of experiments for 35 standard benchmarks. Additionally, the effectiveness and quality of the results are assessed using a real-world case study. Experimental results show that the proposed strategy outperforms one of the competitive strategies, ACTS, for approximately 83% of the benchmarks and achieves similar results to CASA for 65% of the benchmarks when the interaction strength is 2. For an interaction strength of 3, the proposed method outperforms other competitive strategies for approximately 60% and 42% of the benchmarks. The proposed strategy can also generate constrained interaction test suites for an interaction strength of 4, which is not possible for many strategies. The real-world case study shows that the generated test suites can effectively detect injected faults using mutation testing.Software testing plays a significant role in specifying the quality of developed software. High-quality software can be achieved by employing an intensive testing process to detect most of the bugs. Ideally, identifying all possible bugs is achievable by applying exhaustive testing with a large test suite. However, exhaustive testing leads to more time spent in software testing, which increases the development cost. Software that is released without being tested well (due to a release deadline constraint or any other reasons) loses trust in the market. As estimated in [1], software faults can generate a drop in the product stock price of 4 − 6% on average (for companies encountering multiple software failures) and further generate almost 3 billion dollars of market losses. Hence, software testers should gener...

Scalable Path Search for Automated Test Case Generation

“…To ensure the quality of current highly configurable software systems, Hasan et al presented a search-based strategy to generate constrained interaction test suites to cover all possible combinations [7]. The strategy generates the set of all possible t-tuple combinations, and then filters out the set by removing forbidden t-tuples.…”

“…Consequently, our approach does not search the program space in a uniform way, and it works as an intelligent searching technique to some extent to further improve the search quality. The other advantage of adopting the tabu heuristic [7][8][9][10] is to keep the search process from becoming stuck in sub-spaces, as a tabu list can help escape from the local search space.…”

Avocado: Open-Source Flexible Constrained Interaction Testing for Practical Application

“…To this end, the next stage of the varianter algorithm is to optimize the size of the initial set of test cases while maintaining the coverage of all tuples. Here, the CIT varianter uses the mixed neighborhood tabu search (MNTS) algorithm that is newly developed by us [12]. This algorithm is based on the randomized tuning of the solution that leads to a better solution.…”

“…I. Hasan et. al [12] showed a detailed comparison of the test generation algorithm efficiency with other tools.…”