Beginner's luck: a language for property-based generators

Abstract

Property-based random testingà la QuickCheck requires building efficient generators for well-distributed random data satisfying complex logical predicates, but writing these generators can be difficult and error prone. We propose a domain-specific language in which generators are conveniently expressed by decorating predicates with lightweight annotations to control both the distribution of generated values and the amount of constraint solving that happens before each variable is instantiated. This language, called Luck, makes generators easier to write, read, and maintain.We give Luck a formal semantics and prove several fundamental properties, including the soundness and completeness of random generation with respect to a standard predicate semantics. We evaluate Luck on common examples from the property-based testing literature and on two significant case studies, showing that it can be used in complex domains with comparable bug-finding effectiveness and a significant reduction in testing code size compared to handwritten generators.

Composing Bidirectional Programs Monadically

“…One general technique evaluates predicates lazily to drive generation (random or enumerative) [7,9], but one loses control over the resulting distribution of generated values. Luck [15] is a domainspecific language blending narrowing and constraint solving to specify generators as predicates with user-provided annotations to control the probability distribution. In contrast, our programs can be viewed as generators annotated with left inverses with which to derive predicates.…”

Testing noninterference, quickly

“…While the proof of this theorem discovered no errors in the rules or the well-stamped invariant, it did discover a serious flaw in the indistinguishability relation for stacks, which was previously hidden by an error in our stack generator (the wrong definition is described in Section A.5). This illustrates that keeping generators and checkers in sync is challenging and brings further motivation to recent work on domain-specific languages for generators (Claessen et al, 2014;Fetscher et al, 2015;Lampropoulos et al, 2015).…”

Branching processes for QuickCheck generators

“…(In the case that p C v j = 0, the last equation in this proposition holds trivially by (17).) And by replacing (19) in (18) we obtain:…”

“…Recently, Midtgaard et al extend such a technique to test compilers for impure programming languages [23]. Luck [19] is a domain specific language for writing testing properties and QuickCheck generators at the same time. We see Luck's approach as orthogonal to ours, which is mostly intended to be used when we do not know any specific property of the system under test, although we consider that borrowing some functionality from Luck into is an interesting path for future work.…”

“…Feat's authors subsequently extend their approach to uniformly generate values constrained by userdefined predicates [9]. Lastly, Luck is a domain specific language for manually writing QuickCheck properties in tandem with generators so that it becomes possible to finely control the distribution of generated values [19].…”