Industrial control systems are often hybrid systems that are required to satisfy strict performance requirements. Verifying designs against requirements is a difficult task, and there is a lack of suitable open benchmark models to assess, evaluate, and compare tools and techniques. Benchmark models can be valuable for the hybrid systems research community, as they can communicate the nature and complexity of the problems facing industrial practitioners. We present a collection of benchmark problems from the automotive powertrain control domain that are focused on verification for hybrid systems; the problems are intended to challenge the research community while maintaining a manageable scale. We present three models of a fuel control system, each with a unique level of complexity, along with representative requirements in signal temporal logic (STL). We provide results obtained by applying a state of the art analysis tool to these models, and finally, we discuss challenge problems for the research community.

Lyapunov functions are used to prove stability and to obtain performance bounds on system behaviors for nonlinear and hybrid dynamical systems, but discovering Lyapunov functions is a difficult task in general. We present a technique for discovering Lyapunov functions and barrier certificates for nonlinear and hybrid dynamical systems using a searchbased approach. Our approach uses concrete executions, such as those obtained through simulation, to formulate a series of linear programming (LP) optimization problems; the solution to each LP creates a candidate Lyapunov function. Intermediate candidates are iteratively improved using a global optimizer guided by the Lie derivative of the candidate Lyapunov function. The analysis is refined using counterexamples from a Satisfiability Modulo Theories (SMT) solver. When no counterexamples are found, the soundness of the analysis is verified using an arithmetic solver. The technique can be applied to a broad class of nonlinear dynamical systems, including hybrid systems and systems with polynomial and even transcendental dynamics. We present several examples illustrating the efficacy of the technique, including two automotive powertrain control examples.

Many organizations are developing autonomous driving systems, which are expected to be deployed at a large scale in the near future. Despite this, there is a lack of agreement on appropriate methods to test, debug, and certify the performance of these systems. One of the main challenges is that many autonomous driving systems have machine learning components, such as deep neural networks, for which formal properties are difficult to characterize. We present a testing framework that is compatible with test case generation and automatic falsification methods, which are used to evaluate cyber-physical systems. We demonstrate how the framework can be used to evaluate closed-loop properties of an autonomous driving system model that includes the ML components, all within a virtual environment. We demonstrate how to use test case generation methods, such as covering arrays, as well as requirement falsification methods to automatically identify problematic test scenarios. The resulting framework can be used to increase the reliability of autonomous driving systems.

No abstract

Autonomous vehicles are complex systems that are challenging to test and debug. A requirements-driven approach to the development process can decrease the resources required to design and test these systems, while simultaneously increasing the reliability. We present a testing framework that uses signal temporal logic (STL), which is a precise and unambiguous requirements language. Our framework evaluates test cases against the STL formulae and additionally uses the requirements to automatically discover test cases that fail to satisfy the requirements. One of the key features of our tool is the support for machine learning (ML) components in the system design, such as deep neural networks. The framework allows evaluation of the control algorithms, including the ML components, and it also includes models of CCD camera, lidar, and radar sensors, as well as the vehicle environment. We use multiple methods to generate test cases, including covering arrays, which is an efficient method to search discrete variable spaces. The resulting test cases can be used to debug the controller design by identifying controller behaviors that do not satisfy requirements. The test cases can also enhance the testing phase of development by identifying critical corner cases that correspond to the limits of the system's allowed behaviors. We present STL requirements for an autonomous vehicle system, which capture both component-level and systemlevel behaviors. Additionally, we present three driving scenarios and demonstrate how our requirements-driven testing framework can be used to identify critical system behaviors, which can be used to support the development process.