Towards Evaluating Fault Coverage of Protocol Test Sequences

Abstract

In this paper, we investigate the quality of a given protocol test sequence in detecting faulty implementations of a specification. The proposed method differs from the conventional simulation method in that it is not necessary to consider various forms of fault combinations and is guaranteed to identify any faulty machines. Preprocessing and backjumping techniques are used to reduce the computational complexity of the method. We have constructed a tool based on the model and used it in assessing several test sequence generation techniques. Our experience shows that the method is both effective and efficient for even real life protocols.

Timing Fault Models for Systems with Multiple Timers

“…Therefore, edge e j starts both T z and T s : T3 is started by applying i10 within time interval [6,15] After augmentation for both Faults I and V, a correct edge traversal sequence for a non-faulty IUT can be given as: An example test sequence of containing · · · , e 8 , e 9 , e 10 , e 11 , e 12 , · · · is given for the FSM of Figure 6. Suppose the FSM specification defines that, for e 10 , the input i 10 should be applied within time interval of [6,15] seconds (measured from e 8 ) and that e 10 starts T 3 with length D 3 = 4 seconds. Edge e 12 is a timeout transition for T 3 , and for edges e 9 , e 10 , e 11 , and e 12 the costs are c 9 = 4, c 10 = 1, c 11 = 4, and c 12 = 2 seconds, respectively.…”

“…Fault coverage has been studied mostly with respect to transfer/output faults for FSMs [1,9,11,15]. Petrenko et al [9] investigate fundamental underlying concepts of fault coverage analysis, whose primary focus is protocol conformance testing.…”

Guaranteeing full fault coverage for UIO-based testing methods

“…Some, however, considered only the transition identification part in their analysis ( [8], [10], [17]), while others assumed the optimization itself would not interfere with the fault coverage ( [15]). Most of the work on fault coverage evaluation uses the mutation technique: from the specification FSM a certain number of mutant (faulty) FSMs are randomly generated and verified with a given test sequence.…”

“…The number of minimized machines not isomorphic to the real specification FSM gives a measure of the test sequence fault coverage. A more or less similar approach was proposed in [17], where a technique is used to reduce the number of possibilities when reconstructing, from the test sequence, all possible FSMs which would pass the given test sequence. The problem of these "exhaustive" approaches is that even with the reduction techniques the task of enumerating all viable solutions may be still too hard to be feasible in the general case.…”

“…In fact, [SEQ 3] provides full fault coverage, with no need for state identification or reset-preamble sequences, and is much shorter than test sequences generated by UIO based methods. The fault coverage can be verified in this small example by case analysis, or with a tool similar to [16] or [17].…”