Formal Requirement Debugging for Testing and Verification of Cyber-Physical Systems

Abstract

A framework for the elicitation and debugging of formal specifications for Cyber-Physical Systems is presented. The elicitation of specifications is handled through a graphical interface. Two debugging algorithms are presented. The first checks for erroneous or incomplete temporal logic specifications without considering the system. The second can be utilized for the analysis of reactive requirements with respect to system test traces. The specification debugging framework is applied on a number of formal specifications collected through a user study. The user study establishes that requirement errors are common and that the debugging framework can resolve many insidious specification errors 1 . evaluates to true no matter what the system behavior is and, thus, the requirement ϕ is invalid. This is because, if at some time t between 0 and 30 seconds the predicate (v > 100) is false, then the implication (⇒) will trivially evaluate to true at time t and, thus, ϕ will evaluate to true as well. On the other hand, if the predicate (v > 100) is true for all time between 0 and 30 seconds, then the subformula 2 [0,20] (v > 100) will be true at all time between 0 and 10 seconds. This means that the subformula (v > 100) ⇒ 2 [0,20] (v > 100) is true at all time between 0 and 10 seconds. Thus, again, ϕ evaluates to true, which means that ϕ is a tautology.This implies that specification issues are not necessarily artifacts of the graphical user interface and that they can happen even for users who are familiar with temporal logics. Hence, specification elicitation can potentially become an issue as formal and semi-formal testing and verification methods and tools are being adopted by industry. This is because specification elicitation can be performed by untrained users. Therefore, effort can be wasted in checking incorrect requirements, or even worse, the system can pass the incorrect requirements. Clearly, this can lead to a false sense of system correctness, which leads us to the second question: What can be done in an automated way to prevent specification errors in CPS?In this work, we have developed a specification debugging framework to assist in the elicitation of formal requirements. The specification debugging algorithm identifies some of the logical issues in the specifications, but not all of them. Namely, it performs the following: 1) Validity detection: the specification is unsatisfiable or a tautology. 2) Redundancy detection: the formula has redundant conjuncts.3) Vacuity detection: some subformulas do not affect the satisfiability of the formula.Redundancy and vacuity issues usually indicate some misunderstanding in the requirements. As a result, a wide class of specification errors in the elicitation process can be corrected before any test and verification process is initiated. However, some specification issues cannot be detected unless we consider the system, and test the system behaviors with respect to the specification. We provide algorithms to detect specification vacuity with respect to system trace...

Enhancing Temporal Logic Falsification With Specification Transformation and Valued Booleans

“…As has been noted before [34], writing specifications in temporal logic is not trivial. Approaches that have been used to solve this problem are creating tools that make it easier to write specifications [35], automatically detecting faulty specifications [36], and defining template specifications to make it easier for testers to formulate their requirements formally [37]. In this paper, we instead allow test engineers to write specifications in a formalism they already know, namely a causal signal-based framework (in our case using Simulink [13]).…”

Cyber-Physical Systems, a new formal paradigm to model redundancy and resiliency

“…(Aggarwal et al, 1975) state to select the stage where redundancy is to be added, an heuristic criterion is introduced which takes into account the relative increment in reliability versus decrement in performance. (Dokhanchi at al., 2018) propose a framework for the elicitation and debugging of formal specifications for Cyber-Physical Systems through two debugging algorithms. One checks for erroneous or incomplete temporal logic specifications without considering the system and the other can be used for the analysis of reactive requirements with respect to system test traces.…”

MightyL: A Compositional Translation from MITL to Timed Automata

“…C++ code, and the time taken by LTSmin for the actual model-checking); and Uppaal, on all these examples (for the motion planning, only finite words are relevant, hence we report only on the Uppaal running time). We also report on the benchmarks found in [16], where the debugging of formal specifications of cyber-physical systems is reduced to MITL non-satisfiability. More precisely, we check formulae for validity and redundancy.…”

“…More precisely, we check formulae for validity and redundancy. In [16], a formula ϕ is called valid (with respect to a specification goal) if ϕ is neither unsatisfiable nor a tautology, i.e. ϕ and ¬ϕ are both satisfiable.…”