Automata Language Equivalence vs. Simulations for Model-Based Mutant Equivalence: An Empirical Evaluation

Abstract

Mutation analysis is a popular test assessment method. It relies on the mutation score, which indicates how many mutants are revealed by a test suite. Yet, there are mutants whose behaviour is equivalent to the original system, wasting analysis resources and preventing the satisfaction of the full (100%) mutation score. For finite behavioural models, the Equivalent Mutant Problem (EMP) can be addressed through language equivalence of non-deterministic finite automata, which is a wellstudied, yet computationally expensive, problem in automata theory. In this paper, we report on our preliminary assessment of a state-of-the-art exact language equivalence tool to handle the EMP against 3 models of size up to 15,000 states on 1170 mutants. We introduce random and mutation-biased simulation heuristics as baselines for comparison. Results show that the exact approach is often more than ten times faster in the weak mutation scenario. For strong mutation, our biased simulations are faster for models larger than 300 states. They can be up to 1,000 times faster while limiting the error of misclassifying non-equivalent mutants as equivalent to 10% on average. We therefore conclude that the approaches can be combined for improved efficiency.

Mutation Testing Advances: An Analysis and Survey

“…Thus, mutation has been used for almost all forms of testing. Its primary application level is unit testing but several advances have been made in order to support other levels, i.e., specification [2], design [3], integration [4] and system levels [5]. The method has been applied on the most popular programming languages such as C [6], C++ [7], C# [8], Java [9], JavaScript [10], Ruby [11] including specification [2] and modelling languages [12].…”

Model-based mutant equivalence detection using automata language equivalence and simulations

“…This paper extends our previous work [1] on the major following points: the empirical analysis is now performed on 12 models of size up to 15,000 states and 4,710 mutants (instead of 3 models and 1,170 mutants); it adds a new research question to analyse the impact of strong and weak mutation on automata language equivalence performance; finally, we provide statistical significance evidence.…”

Toward Speeding up Mutation Analysis by Memoizing Expensive Methods

“…One main reason behind this problem is that the generated mutants must be tested against the test suite, and usually a large number of mutants are generated, making the process lengthy. Much research has been devoted to reducing the cost of mutation analysis [20], [21], focusing primarily on: (1) reducing the number of generated [22]- [24] or executed mutants [25]- [27];…”