A framework and tool support for the systematic testing of model-based specifications

Abstract

Formal specifications can precisely and unambiguously define the required behaviour of a software system or component. However, formal specifications are complex artifacts that need to be verified to ensure that they are consistent and complete, and validated against the requirements. Current specification testing or animation tools exist to assist with this by allowing the specifier to interpret or execute the specification. However, currently little is known about how to do this effectively.This paper presents a framework and tool support for the systematic testing of formal, modelbased specifications. Several important generic properties that should be satisfied by model-based specifications are first identified. Following the idea of mutation analysis, we then use variants or mutants of the specification to check that these properties are satisfied. The framework also allows the specifier to test application-specific properties. All properties are tested for a range of states that are defined by the tester in the form of a testgraph, which is a directed graph that partially models the states and transitions of the specification being tested. Tool support is provided for the generation of the mutants, for automatically traversing the testgraph and executing the test cases, and for reporting any errors. The framework is demonstrated on a small specification and its application to three larger specifications is discussed. Experience indicates that the framework can be used effectively to test small to medium-sized specifications and that it can reveal a significant number of problems in these specifications.

Design and Implementation of Automated Visualization for Input/Output for Processes in SOFL Formal Specifications

“…The common characteristic of the existing animation techniques is to use test data (also called animation data) to dynamically demonstrate the input-output relation for operations defined in the specification. Compared to the reading and understanding technique, animation is proved to be more effective in validating formal specifications against the client's requirements [3] [4]. However, our experience and study suggest that the effect of specification animation is rather limited due to the fact that input and output data with complex structures can be difficult to comprehend during animation.…”

Integrating Animation-Based Inspection Into Formal Design Specification Construction for Reliable Software Systems

“…An animation approach for Object-Z Specification is described in [62], which simply translates the specification to code for execution. Time Miller and Paul Strooper introduced a framework for animating model-based specifications by using testgraphs [63]. The framework provides a testgraph editor for the user to edit testgraphs, and then derive sequences for animation by traversing the testgraph.…”

A case study in model‐based testing of specifications and implementations

“…Many of the ideas used in this framework are taken from standard software testing to make the framework intuitive for anybody with testing experience. In previous work, the authors have presented a framework and method for specification testing 3, a method for mapping a testgraph to an implementation and using a specification animator as a test oracle 4, and for mapping a testgraph to a GUI for the purpose of implementation testing 5. This paper collates much of this work to present the entire framework for specification testing and implementation testing.…”

“…In the previous work 2, 3, the authors present a framework and method‡ for the systematic testing of formal, model‐based specifications using specification animation. At the core of this framework are testgraphs : directed graphs that partially model the states and transitions of the specification being tested.…”