TESTOR: A Modular Tool for On-the-Fly Conformance Test Case Generation

Abstract

Abstract. We present TESTOR, a tool for on-the-fly conformance test case generation, guided by test purposes. Concretely, given a formal specification of a system and a test purpose, TESTOR automatically generates test cases, which assess using black box testing techniques the conformance to the specification of a system under test. In this context, a test purpose describes the goal states to be reached by the test and enables one to indicate parts of the specification that should be ignored during the testing process. Compared to the existing tool TGV, TESTOR has a more modular architecture, based on generic graph transformation components, is capable of extracting a test case completely on the fly, and enables a more flexible expression of test purposes, taking advantage of the multiway rendezvous. TESTOR has been implemented on top of the CADP verification toolbox, evaluated on three published case-studies and more than 10000 examples taken from the non-regression test suites of CADP.

Reflections on Bernhard Steffen’s Physics of Software Tools

“…Such building blocks have proven successful for the rapid construction of new tools: at present, not less than 94 formal tools 11 have been developed by reusing the software components of CADP. For example, the most recent of these tools is TESTOR [37], which generates conformance tests on the fly and is almost entirely built using three generic technologies of CADP: the BCG environment for on-disk storage of labelled transition systems, the OPEN/CAESAR environment for on-the-fly exploration of labelled transition systems, and the CAESAR SOLVE library for solving Boolean equation systems.…”

Asynchronous Testing of Synchronous Components in GALS Systems

“…We presented an automatic approach integrating both asynchronous and synchronous testing tools to derive complex, but relevant unit test cases for the synchronous components of a GALS system. From a formal model of the system in GRL [25] and a test purpose, the conformance testing tool TESTOR [27] automatically generates a complete test graph [23] capturing the asynchronous behavior of the system relevant to the test purpose. Such a complete test graph is then projected on a synchronous component C and explored using XTL [28] scripts to provide a synchronous test scenario (input constraints in Lutin [34] and an oracle in Lustre [18]) required to test C with the Lurette tool [22].…”

“…Using CADP, we validate the asynchronous aspects of the GRL model, e.g., by checking temporal logic formulas expressing desired (global) correctness properties. Next, we use TESTOR [27] to automatically generate conformance tests, which can be used to assess whether an actual implementation of the GALS system conforms to the GRL model. Then, we project such a conformance test on a synchronous component C and translate it automatically into a scenario (i.e., input constraints in Lutin [34] and an oracle in Lustre [18]), required to automate the testing of C using Lurette [22], the test generation tool of the Lustre V6 toolbox 4 [18,34].…”

Automated Transition Coverage in Behavioural Conformance Testing

“…A popular conformance relation for IOLTSs is ioco [38], which served as basis for various testing approaches, such as on-line testing, as implemented in the JTorX tool [1], or on-the-fly test case generation guided by test purposes, as implemented in the TGV [22] and TESTOR [27] tools. The former approach has the advantage of being fully automatic (the tester executes the SUT and the model in a co-simulation manner), whereas the latter approach using test purposes allows the tester to build a test plan, i.e., set of test purposes at a similar abstraction level as the system requirements.…”

“…Our coverage approach relies on CTG generation and on several automata manipulation features provided by the tools of CADP [13] CADP web page. CTGs are generated using TESTOR [27], a new conformance test case generator that is also able to produce controllable TCs on the fly. The minimal deterministic automata of the CTGs and of the model M are obtained by applying weak trace reduction using the REDUCTOR tool, followed by strong bisimulation minimisation using the BCG MIN tool.…”