Mutation testing with hyperproperties

Abstract

We present a new method for model-based mutation-driven test case generation. Mutants are generated by making small syntactical modifications to the model or source code of the system under test. A test case kills a mutant if the behavior of the mutant deviates from the original system when running the test. In this work, we use hyperproperties—which allow to express relations between multiple executions—to formalize different notions of killing for both deterministic as well as non-deterministic models. The resulting hyperproperties are universal in the sense that they apply to arbitrary reactive models and mutants. Moreover, an off-the-shelf model checking tool for hyperproperties can be used to generate test cases. Furthermore, we propose solutions to overcome the limitations of current model checking tools via a model transformation and a bounded SMT encoding. We evaluate our approach on a number of models expressed in two different modeling languages by generating tests using a state-of-the-art mutation testing tool.

Bounded Model Checking for Hyperproperties

“…HyperQube returns SAT, indicating successful finding of a qualified mutant. We note that in [16] the authors were not able to generate test cases via ϕ mut , as the model checker MCHyper is not able to handle quantifier alternation in push-button fashion.…”

“…Case Study 6: Mutation testing. We adopted the model from [16] and apply the original formula that describes a good test mutant together with the model (see formula ϕ mut in table 3). HyperQube returns SAT, indicating successful finding of a qualified mutant.…”

AutoHyper: Explicit-State Model Checking for HyperLTL

“…HyperLTL [14] is the most studied logic for expressing hyperproperties. A range of problems from different areas in computer science can be expressed as Hyper-LTL MC problems, including (optimal) path panning [38], mutation testing [26], linearizability [30], robustness [21], information-flow control [35], and causality checking [17], to name only a few. Consequently, any model checking tool for HyperLTL is applicable to many disciples within computer science and provides a unified solution to many challenging algorithmic problems.…”

“…In this section, we challenge AutoHyper with complex model checking problems found in the literature. Our benchmarks stem from a range of sources, including non-interference in boolean programs [6], symmetry in mutual exclusion algorithms [18], non-interference in multi-threaded programs [36], fairness in non-repudiation protocols [31], mutation testing [26], and path planning [38].…”

“…Hyperproperties [15] are system properties that relate multiple executions of a system. Such properties are of increasing importance as they naturally occur, e.g., in information-flow control [35], robustness [21], linearizability [29,30], path planning [38], mutation testing [26], and causality checking [17]. A prominent logic to express hyperproperties is HyperLTL, which extends linear-time temporal logic (LTL) with explicit trace quantification [14].…”