Effect of test set minimization on fault detection effectiveness

Abstract

Given a test set T to test a program P, there are at least two attributes of T that determine its fault detection effectiveness. One attribute is the size of T measured as the number of test cases in T. Another attribute is the code coverage measured when P is executed on all elements of T. The fault detection effectiveness of T is the ratio of the number of faults guaranteed to result in program failure when P is executed on T to the total number of faults present in P. An empirical study was conducted to determine the relative importance of the size and coverage attributes in affecting the fault detection effectiveness of a randomly selected test set for some program P. Results from this study indicate that as the size of a test set is reduced, while the code coverage is kept constant, there is little or no reduction in the fault detection effectiveness of the new test set so generated. For the study reported, of the two attributes mentioned above, the code coverage attribute of a test set is more important than its size attribute.* This definition assumes that the underlying hardware does not fail during the execution of a block.

Bi-criteria Test Suite Reduction by Cluster Analysis of Execution Profiles

“…Related work in the context of test suite reduction can be classified into two main categories: The works in which a new technique is presented [3,4,5,16,9,6,7], and empirical studies on the previous techniques [1,2,15,17]. The works which propose a new approach commonly include heuristic algorithms [14], genetic algorithm-based techniques [18] and approaches based on integer linear programming [19].…”

“…In previous empirical studies [15] researchers commonly apply various code coverage criteria in their reduction techniques. The results of empirical studies [2,17] show that the more percentage of the test suite size is reduced the more percentage of faults will be lost.…”

On the effect of test-suite reduction on automatically generated model-based tests

“…In an early study, Wong et al address the question of the effect on fault detection of reducing the size of a test set while holding coverage constant (Wong et al, 1997(Wong et al, , 1998. Their experiments were carried out over a set of commonly used UNIX utilities implemented in C. These programs were manually seeded with faults, producing variant programs each of which contained a single fault.…”

“…The effect on the fault finding capability of the reduced test-suites is, however, unclear and the studies show conflicting evidence. Wong et al (1997Wong et al ( , 1998 found no significant effect in fault finding ability between the full suites and the reduced suites. On the other hand, Rothermel et al (1998) and Jones and Harrold (2003) showed that the reduced test-suites can be dramatically worse with respect to fault finding.…”

“…For example, a reduced test-suite T R may provide the same structural coverage as a test-suite T with significantly fewer test-cases. Previous studies conducted on C code have shown that test-suite reduction techniques significantly reduce the number of test-cases in a test-suite while maintaining the structural coverage of the original suite (Wong et al, 1997(Wong et al, , 1998Rothermel et al, 1998;Jones and Harrold, 2003). The effect on the fault finding capability of the reduced test-suites is, however, unclear and the studies show conflicting evidence.…”