Type Targeted Testing

Abstract

We present a new technique called type targeted testing, which translates precise refinement types into comprehensive test-suites. The key insight behind our approach is that through the lens of SMT solvers, refinement types can also be viewed as a high-level, declarative, test generation technique, wherein types are converted to SMT queries whose models can be decoded into concrete program inputs. Our approach enables the systematic and exhaustive testing of implementations from high-level declarative specifications, and furthermore, provides a gradual path from testing to full verification. We have implemented our approach as a Haskell testing tool called TARGET, and present an evaluation that shows how TARGET can be used to test a wide variety of properties and how it compares against state-of-the-art testing approaches

Beginner's luck: a language for property-based generators

“…1 This approach has been widely investigated, both for generating inputs directly from predicates [13,34,46,66] and for symbolic-executionbased testing [3,9,30,67,70], which additionally uses the system under test to guide generation of inputs that exercise different control-flow paths. For notMember, gathering a set of disequality constraints on x before choosing its value avoids any backtracking.…”

Tests and Proofs for Enumerative Combinatorics

“…In RPBT a random data generator can be defined by filtering the output of another one, in a similar way as an exhaustive generator can be defined by filtering another exhaustive generator in BET. A more generic approach is type-targeted testing [26], wherein types are converted into queries to SMT solvers whose answers provide counterexamples. SmallCheck and Lazy SmallCheck [25] are two Haskell libraries for propertybased testing, allowing an automatic exhaustive testing for small values.…”

Refinement types for TypeScript

“…We envisage a dynamic cast operation cast T :: (x: any) ⇒ {ν : T | ν = x}. It is straightforward to implement cast T for first-order types T as a dynamic check that traverses the value, testing that its components satisfy the refinements [30]. Wrapper-based techniques from the contracts/gradual typing literature should then let us support higher-order types.…”