Coverage guided, property based testing

Abstract

Property-based random testing, exemplified by frameworks such as Haskell's QuickCheck, works by testing an executable predicate (a property) on a stream of randomly generated inputs. Property testing works very well in many cases, but not always. Some properties are conditioned on the input satisfying demanding semantic invariants that are not consequences of its syntactic structureÐe.g., that an input list must be sorted or have no duplicates. Most randomly generated inputs fail to satisfy properties with such sparse preconditions, and so are simply discarded. As a result, much of the target system may go untested. We address this issue with a novel technique called coverage guided, property based testing (CGPT). Our approach is inspired by the related area of coverage guided fuzzing, exemplified by tools like AFL. Rather than just generating a fresh random input at each iteration, CGPT can also produce new inputs by mutating previous ones using type-aware, generic mutation operators. The target program is instrumented to track which control flow branches are executed during a run and inputs whose runs expand control-flow coverage are retained for future mutations. This means that, when sparse conditions in the target are satisfied and new coverage is observed, the input that triggered them will be retained and used as a springboard to go further. We have implemented CGPT as an extension to the QuickChick property testing tool for Coq programs; we call our implementation FuzzChick. We evaluate FuzzChick on two Coq developments for abstract machines that aim to enforce flavors of noninterference, which has a (very) sparse precondition. We systematically inject bugs in the machines' checking rules and use FuzzChick to look for counterexamples to the claim that they satisfy a standard noninterference property. We find that vanilla QuickChick almost always fails to find any bugs after a long period of time, as does an earlier proposal for combining property testing and fuzzing. In contrast, FuzzChick often finds them within seconds to minutes. Moreover, FuzzChick is almost fully automatic; although highly tuned, handwritten generators can find the bugs faster than FuzzChick, they require substantial amounts of insight and manual effort. CCS Concepts: • Software and its engineering → General programming languages.

Generative type-aware mutation for testing SMT solvers

“…The developers promptly fixed this soundness issue in the trunk version of Z3. As we will show (Section 3), Generative Type-Aware Mutation is a powerful generalization of type-aware operator mutation [Winterer et al 2020a] and FuzzChick [Lampropoulos et al 2019]. Neither approach could have generated this bug-triggering formula from the seed 𝜑.…”

“…FuzzChick's Mutator [Lampropoulos et al 2019]: FuzzChick tests Coq programs using grammar-based generators and coverage feedback. Adapted to our setting, this corresponds to 𝜑 mutant = 𝜑 [expr 2 /expr 1 ] where expr 1 , expr 2 are expressions from expr (𝜑).…”

“…Relationships to Type-Aware Operator Mutation and FuzzChick. Generative Type-Aware Mutation generalizes two fuzzing techniques: type-aware operator mutation [Winterer et al 2020a], and FuzzChick [Lampropoulos et al 2019]. Type-aware operator mutation mutates the operators within the SMT-LIB formulas making use of operators from the SMT-LIB specification.…”

“…Mutation-based fuzzing techniques leverage user-provided test seeds and generate new mutated inputs to uncover bugs in programs. The two closest works from the family of mutation-based testing techniques are skeletal program enumeration (SPE) for testing C compilers[Zhang et al 2017], and FuzzChick[Lampropoulos et al 2019] an approach to test Coq programs. Similar to generative type-aware mutation, SPE also performs random type-aware mutations.…”

Do Judge a Test by its Cover

“…Coverage-guided fuzzing tools like AFL [22] can be viewed as a way of using a different form of feedback (branch instead of combinatorial coverage) to improve the generation of random inputs by finding more "interesting" tests. Fuzzing is a huge topic [43] that has exploded in popularity recently, with researchers evaluating the benefits of using more forms of feedback [13,31], incorporating learning [28,33] or symbolic [39,42] techniques, and bringing the benefits of these methods to functional programming [11,21]. One fundamental difference, however, is that all of these techniques are online and grey-box: they instrument and execute the program on various inputs in order to obtain feedback.…”