Evaluating Different Strategies for Reduction of Mutation Testing Costs

Lima, Jackson A. Prado; Guizzo, Giovani; Vergílio, Silvia Regina; Silva, Alan P. C.; Filho, Helson Luiz Jakubovski; Ehrenfried, Henrique Varella

doi:10.1145/2993288.2993292

Cited by 15 publications

(10 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…b We are now executing the mutant index with value 3 that is the third mutant proceeding from b + c. This value has been saved in the aforementioned file and is assigned to currentMutant in the loadCurrentMutant method. This value (3) is searched in the array passed in the variable parameters set (it was in previous studies [1,2,24,44]) and found with location = 2. Then, the method returns a/b.…”

Section: Mutant Schema Generationmentioning

confidence: 99%

Analysing the combination of cost reduction techniques in Android mutation testing

Usaola

Rodríguez-Trujillo

2021

Software Testing Verif & Rel

View full text Add to dashboard Cite

When applied to mobile software, mutation testing is particularly costly due to the deployment of the app under test onto the device: if one deployment is made for each generated mutant, the execution time becomes unapproachable. This paper analyses how the combination of different cost reduction techniques improves the execution time of mutation testing in mobile apps. The techniques reviewed and combined are mutant schema, parallel execution and two different ways of executing tests against the mutants (all against all and all against mutants remaining alive), as well as greedy algorithm for reducing the test suite size. This paper also presents a mathematical model of cost reduction and checks its validity with several experiments. Furthermore, the exhaustive and long experimentation has led the authors to compile a set of good practices which are also presented in a set of lessons learned.

show abstract

Section: Mutant Schema Generationmentioning

confidence: 99%

Analysing the combination of cost reduction techniques in Android mutation testing

Usaola

Rodríguez-Trujillo

2021

Software Testing Verif & Rel

View full text Add to dashboard Cite

show abstract

“…RMS is arguably the most naive type of mutant reduction approach, as mutants are discarded with no particular preference, thus representing an unbiased reduction. This strategy has also served as a baseline comparison for several other work in the literature [24,53,94,97]. Therefore, assessing the error for this type of strategy provides a meaningful overview of what to expect from mutant reduction strategies in general.…”

Section: Rq1 ś Random Mutant Samplingmentioning

confidence: 99%

“…As we showed in Section 2, 60 out of 75 evaluated papers use either one or the other. Furthermore, they both are commonly used as a baseline for comparison in mutant reduction work [24,53,94,97].…”

Section: Rq3 ś Strategy Differencesmentioning

confidence: 99%

“…We limited our experimentation to RMS and SM strategies. We decided to study those two strategies because they are two of the most common mutant reduction strategies in the literature [41,67,75] and because they are commonly used as a baseline for newly proposed strategies [53,59,94,95]. In fact, 60 out of 75 papers we surveyed on mutant reduction (Section 2) use either RMS or SM, and 15 of them use both.…”

Section: Threats To Validitymentioning

confidence: 99%

See 1 more Smart Citation

Cost measures matter for mutation testing study validity

Guizzo

Sarro

Harman

2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

Self Cite

View full text Add to dashboard Cite

Mutation testing research has often used the number of mutants as a surrogate measure for the true execution cost of generating and executing mutants. This poses a potential threat to the validity of the scientific findings reported in the literature. Out of 75 works surveyed in this paper, we found that 54 (72%) are vulnerable to this threat. To investigate the magnitude of the threat, we conducted an empirical evaluation using 10 real-world programs. The results reveal that: i) percentages of randomly sampled mutants differ from the true execution time, on average, by 44%, varying in difference from 19% to 91%; ii) errors arising from using the surrogate correlate with program size (= 0.74) and number of mutants (= 0.76), making the problem more pernicious for more realistic programs; iii) scientific findings concerning sampling strategies would have approximately 37% rank disagreement, indicating potentially dramatic impact on experiment validity. To investigate whether this threat matters in practice, we reproduced a seminal study on Selective Mutation (widely relied upon for more than two decades). The impact is stark: an inconclusive scientific finding using the surrogate is transformed to an unequivocal finding when using the true execution cost. CCS CONCEPTS • Software and its engineering → Software testing and debugging; • General and reference → Empirical studies; Measurement.

show abstract

“…Harman et al [37] provided a more restrictive fitness function to find strongly subsuming high order mutants (SSHOMs). Lima et al [38] recently compared HOM-based strategies and traditional strategies, such as random mutant selection, selective mutation and search-based mutant selection (genetic algorithm). This comparison was based on the number of mutants, the number of test cases and the mutation score.…”

Section: Related Workmentioning

confidence: 99%

Search-based mutant selection for efficient test suite improvement: Evaluation and results

Delgado‐Pérez

Medina‐Bulo

2018

Information and Software Technology

View full text Add to dashboard Cite

Context: Search-based techniques have been applied to almost all areas in software engineering, especially to software testing, seeking to solve hard optimization problems. However, the problem of selecting mutants to improve the test suite at a lower cost has not been explored to the same extent as other problems, such as mutant selection for test suite evaluation or test data generation. Objective: In this paper, we apply search-based mutant selection to enhance the quality of test suites efficiently. Namely, we use the technique known as Evolutionary Mutation Testing (EMT), which allows reducing the number of mutants while preserving the power to refine the test suite. Despite reported benefits of its application, the existing empirical results were derived from a limited number of case studies, a particular set of mutation operators and a vague measure, which currently makes it difficult to determine the real performance of this technique. Method: This paper addresses the shortcomings of previous studies, providing a new methodology to evaluate EMT on the basis of the actual improvement of the test suite achieved by using the evolutionary strategy. We make use of that methodology in new experiments with a carefully selected set of real-world C++ case studies. Results: EMT shows a good performance for most case studies and levels of demand of test suite improvement (around 45% less mutants than random selection in the best case). The results reveal that even a reduced subset of mutants selected with EMT can serve to increase confidence in the test suite, especially in programs with a large set of mutants. Conclusions: These results support the use of search-based techniques to solve the problem of mutant selection for a more efficient test suite refinement. Additionally, we identify some aspects that could foreseeably help enhance EMT.

show abstract

Evaluating Different Strategies for Reduction of Mutation Testing Costs

Cited by 15 publications

References 41 publications

Analysing the combination of cost reduction techniques in Android mutation testing

Analysing the combination of cost reduction techniques in Android mutation testing

Cost measures matter for mutation testing study validity

Search-based mutant selection for efficient test suite improvement: Evaluation and results

Contact Info

Product

Resources

About