Search-based mutant selection for efficient test suite improvement: Evaluation and results

Abstract

Context: Search-based techniques have been applied to almost all areas in software engineering, especially to software testing, seeking to solve hard optimization problems. However, the problem of selecting mutants to improve the test suite at a lower cost has not been explored to the same extent as other problems, such as mutant selection for test suite evaluation or test data generation. Objective: In this paper, we apply search-based mutant selection to enhance the quality of test suites efficiently. Namely, we use the technique known as Evolutionary Mutation Testing (EMT), which allows reducing the number of mutants while preserving the power to refine the test suite. Despite reported benefits of its application, the existing empirical results were derived from a limited number of case studies, a particular set of mutation operators and a vague measure, which currently makes it difficult to determine the real performance of this technique. Method: This paper addresses the shortcomings of previous studies, providing a new methodology to evaluate EMT on the basis of the actual improvement of the test suite achieved by using the evolutionary strategy. We make use of that methodology in new experiments with a carefully selected set of real-world C++ case studies. Results: EMT shows a good performance for most case studies and levels of demand of test suite improvement (around 45% less mutants than random selection in the best case). The results reveal that even a reduced subset of mutants selected with EMT can serve to increase confidence in the test suite, especially in programs with a large set of mutants. Conclusions: These results support the use of search-based techniques to solve the problem of mutant selection for a more efficient test suite refinement. Additionally, we identify some aspects that could foreseeably help enhance EMT.

Evaluation of alternative design choices for evolutionary mutation testing by means of automated configuration

“…In this improved version, first, a subset of operators is selected at random until reaching the stopping condition; second, in order to discard as many useless mutants as possible, mutants are selected at random from that subset of operators until reaching the stopping condition again. This version showed more competitive results than conventional selective mutation and random selection in previous experiments assessing EMT's effectiveness (Delgado-Pérez and Medina-Bulo, 2018). The figure shows the average and standard deviation of the percentage of all the mutants that are required by each technique in the programs under test.…”

“…In the first studies, EMT was evaluated in the context of web service compositions (Domínguez-Jiménez et al, 2011). The assessment of this technique was later extended to other contexts, i.e., object-oriented programs Delgado-Pérez and Medina-Bulo, 2018) and event processing queries (Gutiérrez-Madroñal et al, 2019), providing evidence that this approach could improve over other known selection techniques, such as mutant sampling (Budd, 1980) and selective mutation (Barbosa et al, 2001). In a related paper, Schwarz et al (2011) also applied a GA to identify undetected mutants for the test suite improvement, where they considered the impact that mutations had on the code coverage and how these mutations were distributed over the code, thus increasing the number of useful mutants selected in their experiments.…”

“…Taking into account the aforementioned problems, this means that we might have underestimated the effectiveness of EMT -especially, if each particular context requires different parameter values or if an optimal configuration was not found in the original experiments-. In fact, all experiments applying EMT after that , (Delgado-Pérez and Medina-Bulo, 2018), (Gutiérrez-Madroñal et al, 2019) have followed the presumably best parameter settings reached in that paper (Domínguez-Jiménez et al, 2011). This is because restarting the same experiments to find a tailored combination in each new context is tedious and costly, especially if the parameter tuning has to be done completely by hand.…”

“…With the aim to resolve this trade-off, different techniques have been proposed in the past, such as mutant sampling (random selection of a subset of mutants, Budd (1980)), selective mutation, (selection of a subset of mutation operators, Barbosa et al (2001)) or higher order mutation (combination of two or more mutations in the same mutant, Jia and Harman (2008)). In this paper, we focus on evolutionary mutation testing (EMT) (Delgado-Pérez and Medina-Bulo, 2018), a search-based technique that uses a genetic algorithm (GA) to favor the selection of mutants with the potential to assist a tester in the refinement of a test suite (i.e., mutants that remain alive). Notably, this technique has shown to be able to outperform mutant sampling and selective mutation (Domínguez-Jiménez et al, 2011;Delgado-Pérez and Medina-Bulo, 2018) in previous experimental procedures.…”

“…In this paper, we focus on evolutionary mutation testing (EMT) (Delgado-Pérez and Medina-Bulo, 2018), a search-based technique that uses a genetic algorithm (GA) to favor the selection of mutants with the potential to assist a tester in the refinement of a test suite (i.e., mutants that remain alive). Notably, this technique has shown to be able to outperform mutant sampling and selective mutation (Domínguez-Jiménez et al, 2011;Delgado-Pérez and Medina-Bulo, 2018) in previous experimental procedures.…”