Some Seeds Are Strong: Seeding Strategies for Search-based Test Case Selection

Abstract

The time it takes software systems to be tested is usually long. Search-based test selection has been a widely investigated technique to optimize the testing process. In this paper, we propose a set of seeding strategies for the test case selection problem that generate the initial population of pareto-based multi-objective algorithms, with the goals of (1) helping to find an overall better set of solutions and (2) enhancing the convergence of the algorithms. The seeding strategies were integrated with four state-of-the-art multi-objective search algorithms and applied into two contexts where regression-testing is paramount: (1) Simulation-based testing of Cyber-Physical Systems and (2) Continuous Integration. For the first context, we evaluated our approach by using six fitness function combinations and six independent case studies, whereas in the second context we derived a total of six fitness function combinations and employed four case studies. Our evaluation suggests that some of the proposed seeding strategies are indeed helpful for solving the multi-objective test case selection problem. Specifically, the proposed seeding strategies provided a higher convergence of the algorithms towards optimal solutions in 96% of the studied scenarios and an overall cost-effectiveness with a standard search budget in 85% of the studied scenarios.

Enhancing Multi-Objective Test Case Selection through the Mutation Operator

“…The single-objective mutation operator only considers the test adequacy to obtain the selection and removal probabilities for each of the test cases. However, this poses a key disadvantage in those cases in which the execution time of test cases plays a crucial role, such as the case of cyber-physical systems (CPSs) [7,16]. To this end, in this paper we extend the single-objective mutation operator, and instead, consider multiple objectives: one related to effectiveness (i.e., adequacy score) and the second one related to cost (i.e., test execution time).…”

“…The dataset involves a total of 6 Simulink models (encompassing each a different CPS case study) of different characteristics and complexities (i.e., in terms of size, number of inputs and outputs). In addition to the four case study systems used in our conference version paper [26], we used two additional case study systems, namely, Cruise Controller (CC) and Tiny.Table 1 summarizes the main characteristics of the selected Simulink models. Each Simulink model encompassed between 120 to 150 test cases.…”

“…Both, in the case of the performance of the search algorithm as well as in the case of the mutation operator, the fitness functions play an important role. Because of this, we considered results from previous studies to select the most cost-effective fitness functions [16,33]. As for the cost function, in all the three cases we selected the test execution time, as it brings a more coarse-grain measure of the cost of executing each test case.…”

“…Search algorithms have been found to be effective in solving multiple software engineering tasks, including test case generation [1][2][3][4][5], fault localization [6], and regression test optimization [7][8][9][10][11][12][13][14][15]. Because of the time it requires for test cases to execute, in the last few years, significant effort has been devoted to the field of test case selection [7,12,13,[15][16][17]. Test case selection aims at reducing the number of test cases to execute, while maintaining as high as possible the overall test quality.…”

“…Despite all the work in the field of search-based test case selection, to the best of our knowledge, there is no study that has investigated how to enhance the test case selection problem from the mutation operator perspective. Indeed, most of the studies employ the default value of bit-flip mutation with a probability of 1/N , N being the number of test cases in the initial test suite [12,13,16,18,[21][22][23].…”