Relating counterexamples to test cases in CTL model checking specifications

Abstract

Counterexamples produced by model checkers are frequently exploited for the purpose of testing. Counterexamples and test cases are generally treated as essentially the same thing, while in fact they can differ significantly. For example, it might take more than one test case to "cover" a given counterexample, because not all property violations can be illustrated with linear counterexamples. This paper presents a formal relationship between counterexamples and test cases in the context of the Computation Tree Logic (CTL), the logic of the popular model checker SMV. Given a test requirement as a CTL formula, we define what it means for a set of test cases to cover a counterexample associated with that requirement. This result can not only be used in the generation of a test set that satisfies a given test coverage criterion, but also in the determination of whether an extant test set satisfies the criterion. Our results can guide the production of counterexamples in model checkers explicitly intended to support testing.

Testing with model checkers: a survey

“…In a related work, Wijesekera et al [30] define a formal relation between test cases and counterexamples for full CTL. This relation is described in Section 7.8.…”

“…MC/DC, for example, requires pairs of test cases to cover conditions. As shown in [30], such requirements can be expressed with CTL. However, current model checkers do not support full CTL but only a linear subset such as such as ACT L det or LIN [28].…”

“…However, current model checkers do not support full CTL but only a linear subset such as such as ACT L det or LIN [28]. To overcome this problem, it is proposed in [30] that model checkers do not generate linear traces but a tree like structure called evidence graphs. Evidence graphs can be nonlinear, and illustrate for any CTL formula, why it is satisfied or violated.…”

“…This is simply achieved by creating one test case for each path from the root node to a leaf node of the evidence graph. The approach described in [30] is not implemented. Clarke et al [29] proposed an algorithm to create tree-like counterexamples, which would serve a similar purpose; at the time of this writing, however, there is no available implementation.…”

Graded CTL Model Checking for Test Generation

“…A good choice of the test suite is the key for successful deductions of faults in simulation processes. It is now more than a decade that model-checking is used for this purpose, see [10,15,4,2,3,11]. In this context, a high level abstraction of the System Under Test (SUT), is necessary.…”