Static detection of equivalent mutants in real-time model-based mutation testing

Abstract

Model-based mutation testing has the potential to effectively drive test generation to reveal faults in software systems. However, it faces a typical efficiency issue since it could produce many mutants that are equivalent to the original system model, making it impossible to generate test cases from them. We consider this problem when model-based mutation testing is applied to real-time system product lines, represented as timed automata. We define novel, time-specific mutation operators and formulate the equivalent mutant problem in the frame of timed refinement relations. Further, we study in which cases a mutation yields an equivalent mutant. Our theoretical results provide guidance to system engineers, allowing them to eliminate mutations from which no test case can be produced. Our empirical evaluation, based on a proof-of-concept implementation and a set of benchmarks from the literature, confirms the validity of our theory and demonstrates that in general our approach can avoid the generation of a significant amount of the equivalent mutants.

MUPPAAL: Reducing and Removing Equivalent and Duplicate Mutants in UPPAAL

“…Preventing and removing such useless mutants reduces the computation costs of (MB)MT and builds more trust in mutation scores. Recently, Basile et al tackled the equivalent mutant problem for Timed Automata with Input and Output (TAIO): they defined mutation operators preventing mutants from refining the original system [6], [5]. However, this technique does not address duplicate mutants: in our experiments, up to 32% of all generated mutants were duplicates.…”

“…: A duplicate mutant has the same behaviour as another mutant and is thus useless. Basile et al refinement technique ensures nonequivalence [6], [5] but does not avoid mutant duplicates. Hence, we introduce a new mutation operator (SMI-NR), avoiding first-order duplicate mutants between SMI and TMI.…”

“…These mutation operators follow the same construction as those defined in [2], [21]. However, Basile et al use a timed refinement technique to avoid the generation of equivalent mutants but do not address mutant duplicates [5], [6]. Larsen et al defined an MBMT technique [29] on top of the UPPAAL-ECDAR verification tool [11].…”

“…The heuristic suggests a pair of mutants as duplicates otherwise; 4) We implemented MUPPAAL using the UPPAAL execution engine for TAs and UPPAAL-TRON [30] for timed trace generation and checking. In addition to the above contributions, MUPPAAL supports mutation operators for TA from [1], [6], [35] avoiding equivalent mutants using refinement prevention [5], [6]; 5) We assessed MUPPAAL 1 . Our results on four cases indicate that timed bisimulation offers the best tradeoff between performance and accuracy of detection.…”