Effective Hybrid System Falsification Using Monte Carlo Tree Search Guided by QB-Robustness

Abstract

Hybrid system falsification is an important quality assurance method for cyber-physical systems with the advantage of scalability and feasibility in practice than exhaustive verification. Falsification, given a desired temporal specification, tries to find an input of violation instead of a proof guarantee. The state-of-the-art falsification approaches often employ stochastic hill-climbing optimization that minimizes the degree of satisfaction of the temporal specification, given by its quantitative robust semantics. However, it has been shown that the performance of falsification could be severely affected by the so-called scale problem, related to the different scales of the signals used in the specification (e.g., rpm and speed): in the robustness computation, the contribution of a signal could be masked by another one. In this paper, we propose a novel approach to tackle this problem. We first introduce a new robustness definition, called QB-Robustness, which combines classical Boolean satisfaction and quantitative robustness. We prove that QB-Robustness can be used to judge the satisfaction of the specification and avoid the scale problem in its computation. QB-Robustness is exploited by a falsification approach based on Monte Carlo Tree Search over the structure of the formal specification. First, tree traversal identifies the sub-formulas for which it is needed to compute the quantitative robustness. Then, on the leaves, numerical hill-climbing optimization is performed, aiming to falsify such sub-formulas. Our in-depth evaluation on multiple benchmarks demonstrates that our approach achieves better falsification results than the state-of-the-art falsification approaches guided by the classical quantitative robustness, and it is largely not affected by the scale problem.

Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions

“…We implement ATheNA-S as a plugin for S-Taliro [10], an opensource SBST tool. We selected S-Taliro, among other alternatives (e.g., Breach [26], FalCAuN [94], falsify [97], FalStar [31,93], Fore-See [98]) due to its recent classification as ready for industrial development [48], and its use in several industrial systems (e.g., [90]). In addition, this choice makes our solution applicable with other S-Taliro plugins, such as Aristeo [67].…”

Property-Based Mutation Testing

“…ART is a baseline strategy that generates evenly distributed test cases (within valid input ranges), thereby ensuring adequate diversity in the test inputs. On the other hand, FT generates counterexamples i.e., test cases that violate a property for a given model [38], [39]. Note that ART and FT work in radically complementary ways.…”

STLmc: Robust STL Model Checking of Hybrid Systems Using SMT

“…There exist many tools for falsifying STL properties of hybrid systems, including Breach [14], S-talrio [1], and TLTk [11]. STL falsification techniques are based on STL monitoring [13,32], and often use stochastic optimization techniques, such as Ant-Colony Optimization [1], Monte-Carlo tree search [43], deep reinforcement learning [41], and so on. These techniques are often quite useful for finding counterexamples in practice, but, as mentioned, cannot be used to verify STL properties of hybrid systems.…”

“…Due to the infinite-state nature of hybrid systems with continuous dynamics, most techniques and tools for analyzing STL properties focus on monitoring and falsification. These techniques analyze concrete samples of signals obtained by simulating hybrid automata to monitor the system's behavior [13,15,32] or find counterexamples [1,37,43], often combined with stochastic optimization. To this end, STL monitoring and falsification use quantitative semantics that defines the robustness degree to indicate how well the formula is satisfied.…”