Test-case reduction for C compiler bugs

Abstract

To report a compiler bug, one must often find a small test case that triggers the bug. The existing approach to automated test-case reduction, delta debugging, works by removing substrings of the original input; the result is a concatenation of substrings that delta cannot remove. We have found this approach less than ideal for reducing C programs because it typically yields test cases that are too large or even invalid (relying on undefined behavior). To obtain small and valid test cases consistently, we designed and implemented three new, domain-specific test-case reducers. The best of these is based on a novel framework in which a generic fixpoint computation invokes modular transformations that perform reduction operations. This reducer produces outputs that are, on average, more than 25 times smaller than those produced by our other reducers or by the existing reducer that is most commonly used by compiler developers. We conclude that effective program reduction requires more than straightforward delta debugging.

Automatic testing of symbolic execution engines via program generation and differential testing

“…Our approach takes advantage of the recent work on testing compilers [22], [23], [34], especially the work of Regehr et al on program generation [34] and reduction [27]. Combined with our technique for generating program variants in three different modes and a set of appropriate oracles, we show that these techniques are effective at finding bugs in symbolic execution engines.…”

“…These different versions are run and crosschecked using our four oracle types: crash detection, and output, function calls and coverage comparison (Employ oracles). Any programs exposing mismatches (as flagged by our oracles) between the native and symbolic execution runs (Gather mismatches) are then reduced using the C-Reduce tool [27] (Reduce programs) and reported to developers.…”

“…Debugging such programs would be highly difficult. Therefore, for each generated program that exposes a bug, we used the C-Reduce tool [27] to reduce it to a manageable size.…”

“…Finally, whenever a generated program finds a bug in the symbolic execution tool, we use existing program reduction techniques [27] which are combined with our oracles in order to obtain a small program (with fewer than 30 lines of code in our experiments) that forms an easy-to-understand, reproducible bug report.…”

ORBS and the limits of static slicing

“…Delta [34] uses a separate tool to flatten tree structures found in programs before applying delta debugging. Regehr et al [39] exploit the syntax and semantics of C for four delta-debugging based algorithms to minimize C programs that trigger compiler bugs.…”