Generating automated and online test oracles for Simulink models with continuous and uncertain behaviors

Abstract

Test automation requires automated oracles to assess test outputs. For cyber physical systems (CPS), oracles, in addition to be automated, should ensure some key objectives: (i) they should check test outputs in an online manner to stop expensive test executions as soon as a failure is detected; (ii) they should handle time-and magnitude-continuous CPS behaviors; (iii) they should provide a quantitative degree of satisfaction or failure measure instead of binary pass/fail outputs; and (iv) they should be able to handle uncertainties due to CPS interactions with the environment. We propose an automated approach to translate CPS requirements specified in a logic-based language into test oracles specified in Simulink -a widely-used development and simulation language for CPS. Our approach achieves the objectives noted above through the identification of a fragment of Signal First Order logic (SFOL) to specify requirements, the definition of a quantitative semantics for this fragment and a sound translation of the fragment into Simulink. The results from applying our approach on 11 industrial case studies show that: (i) our requirements language can express all the 98 requirements of our case studies; (ii) the time and effort required by our approach are acceptable, showing potentials for the adoption of our work in practice, and (iii) for large models, our approach can dramatically reduce the test execution time compared to when test outputs are checked in an offline manner.

The virtual lands of Oz: testing an agribot in simulation

“…Rather than reporting a Pass/Fail verdict, the oracle may report a degree of satisfaction or violation of the checked requirements. Such a grading approach has been formalized by recent work on cyber-physical systems (Menghi et al, 2019), for properties expressed in a fragment of the Signal First Order logic. For example, if an output signal should remain below a threshold, a slight overshooting is graded as less severe than a large one.…”

A Tool for the Automatic Generation of Test Cases and Oracles for Simulation Models Based on Functional Requirements

“…In the context of test oracle generation for simulationbased testing there is much less work as compared with test case generation [21]. Our previous tool generated test systems for simulation-based testing of configurable CPS [22], which included test cases and oracles.…”

Evaluating model testing and model checking for finding requirements violations in Simulink models

“…Dually, the max operator captures ∨. [27] into quantitative fitness functions to be used in the model testing approach in Figure 1(a) Translation to robustness metric [19,34]…”

“…. , s n } of signals at time t, we have R (S,t ) (φ) ≥ 0 if and only if φ holds over the set S at time t [19,34]. That is, we can infer boolean satisfaction of STL formulas based on their fitness values computed by R. In Table 3, ϵ is an infinitesimal positive value that is used to ensure the above relation between boolean satisfiability and fitness values of real-valued constraints (i.e., µ rel-op 0) and literals (i.e., ⊤, ⊥, and s B ) in the STL grammar.…”