Coverage and Its Discontents

Abstract

Everyone wants to know one thing about a test suite: will it detect enough bugs? Unfortunately, in most settings that matter, answering this question directly is impractical or impossible. Software engineers and researchers therefore tend to rely on various measures of code coverage (where mutation testing is considered a form of syntactic coverage). A long line of academic research efforts have attempted to determine whether relying on coverage as a substitute for fault detection is a reasonable solution to the problems of test suite evaluation. This essay argues that the profusion of coverage-related literature is in part a sign of an underlying uncertainty as to what exactly it is that measuring coverage should achieve, as well as how we would know if it can, in fact, achieve it. We propose some solutions and mitigations, but the primary focus of this essay is to clarify the state of current confusions regarding this key problem for effective software testing.

Distance-Integrated Combinatorial Testing

“…On the one hand, the effectiveness of t-way testing has been shown by a number of empirical studies [4], [17], [20], [27], [30], [42]. On the other hand, coverage-based software testing could raise an open question whether the coverage is actually useful for real fault detection [21]. Evaluating the improvement of fault detection effectiveness by DICOT is further work.…”

Cause reduction: delta debugging, even without bugs

“…In fact, the unprioritized Full suite was the worst 30‐s quick test for statement and branch coverage, and the third best for 5 m, but killed the most mutants for 5‐m tests, and performed better than all but the Δ prioritized suites for 30 s. This somewhat unexpected result merits examination in future work: perhaps coverage alone is particularly weak in its ability to produce varied behavior in YAFFS2, because of the complex state of the file system itself, and the advantage longer tests have in producing complex state . Some other peculiar interaction of coverage and mutation testing (because there are not enough real YAFFS2 faults to investigate) may also be involved. For now, it seems that while cause reduction is not ideal for YAFFS2 quick testing, the basic preservation of most properties holds even here.…”

Ensuring the Observability of Structural Test Obligations

“…Adequacy criteria have seen widespread use, as they offer objective, measurable checklists [26] and-importantly-stopping criteria for the testing process. For that same reason, they are ideal as test generation targets [52,44,35], as coverage can be straightforwardly measured and optimized for [5].…”