An Integrated Approach to Assertion-Based Random Testing in Prolog

Abstract

We present an approach for assertion-based random testing of Prolog programs that is tightly integrated within an overall assertionbased program development scheme. Our starting point is the Ciao model, a framework that unifies unit testing and run-time verification, as well as static verification and static debugging, using a common assertion language. Properties which cannot be verified statically are checked dynamically. In this context, the idea of generating random test values from assertion preconditions emerges naturally since these preconditions are conjunctions of literals, and the corresponding predicates can in principle be used as generators. Our tool generates valid inputs from the properties that appear in the assertions shared with other parts of the model, and the run time-check instrumentation of the Ciao framework is used to perform a wide variety of checks. This integration also facilitates the combination with static analysis. The generation process is based on running standard predicates under non-standard (random) search rules. Generation can be fully automatic but can also be guided or defined specifically by the user. We propose methods for supporting (C)LP-specific properties, including combinations of shape-based (regular) types and variable sharing and instantiation, and we also provide some ideas for shrinking for these properties. We also provide a case study applying the tool to the verification and checking of the code of some of the abstract domains used by the Ciao system. Research partially funded by MINECO TIN2015-67522-C3-1-R TRACES project, and the Madrid P2018/TCS-4339 BLOQUES-CM program. We are also grateful to the anonymous reviewers for their useful comments. At run time, sorted(Ys) will be called within the assertion checking-harness, right after calls to qs/2. This harness ensures that the variable bindings (or constraints) and the whole checking process are kept isolated from the normal execution of the program (this can be seen conceptually as including a Prolog copy_term, or calling within a double negation, \+\+, executing in a separate process, etc.). Testing vs. Run-Time Checking: The checking of sorted/1 in the example above will occur in principle during execution of the program, i.e., in deployment. However, in many cases it is not desirable to wait until then to detect errors. This is the case for example if errors can be catastrophic or perhaps if there is interest in testing, perhaps for debugging purposes, more general properties that have not been formally proved and whose main statements are not directly part of the program (and thus, will never be executed), such as, e.g.:

Analysis and Transformation of Constrained Horn Clauses for Program Verification

“…In particular, the Ciao assertion framework (Hermenegildo et al 1999; implements assertionbased testing: the properties that appear in assertions are defined using predicates, and then the preconditions of such assertions act as generators that are used to drive the run-time testing of those parts of assertions that are not discharged at compile time, essentially embodying the PBT approach. Recent work (Casso et al 2019) shows how this generation process can be performed for complex properties and random values by executing the predicates defining such properties under different search rules (e.g. breadth-first, iterative deepening, random), available in the Ciao system.…”

Testing Your (Static Analysis) Truths

“…However, the Ciao model also includes mechanisms for generating test cases automatically from the assertion preconditions, using the corresponding property predicates as generators. This has been extended recently [10] to a random test case generation framework, which automatically generates, using the same technique, random test cases that satisfy assertion preconditions. We refer to the combination of this test generation mechanism with the run-time checking of the intervening assertions as assertion-based testing, that is, generating and running relevant test cases which exercise the run-time checks of the assertions in a program, thus testing if those assertions are correct.…”

“…Test Case Generation. In the absence of entry assertions, the test case generation framework [10] has already some mechanisms to generate relevant test cases, instead of random, nonsensical inputs which would exercise few run-time checks before failing. However, these generators have limitations, and the assertionbased testing framework is in fact best used with assertions that have descriptiveenough call patterns, or with custom user-defined generators in their absence.…”

“…While the essential parts, such as the fixpoint algorithms and the classic abstract domains, have been used routinely for a long time now and it is unusual to find bugs, other parts are less developed and yet others are prototypes or even proofs of concept. A recent, shallow effort of applying a new testing tool to some parts of the Ciao analyzers as a case study [10] revealed subtle bugs, not only in the less-developed parts of the system, but also in corner cases of the parts that are considered more mature, such as, e.g., in the handling of rarely-used built-ins.…”

“…Another feature of Ciao that will be instrumental to our approach is the use of a unified assertion language and framework across its different components [22,23], which together implement its unique blend of static and dynamic assertion checking. These components include: 1) the PLAI static analyzer [19,25,40], which expresses the inferred information as Ciao assertions interspersed within the original program; 2) the assertion runtime-checking framework [45,46], which instruments the code to ensure that any assertions remaining after static verification are not violated at run time; 3) the (random) test case generation framework [10], which generates random test cases satisfying the properties present in an assertion preconditions; 4) the unit-test framework [36], which executes those test cases.…”