Testing advanced driver assistance systems using multi-objective search and neural networks

Abstract

Recent years have seen a proliferation of complex Advanced Driver Assistance Systems (ADAS), in particular, for use in autonomous cars. These systems consist of sensors and cameras as well as image processing and decision support software components. They are meant to help drivers by providing proper warnings or by preventing dangerous situations. In this paper, we focus on the problem of design time testing of ADAS in a simulated environment. We provide a testing approach for ADAS by combining multiobjective search with surrogate models developed based on neural networks. We use multi-objective search to guide testing towards the most critical behaviors of ADAS. Surrogate modeling enables our testing approach to explore a larger part of the input search space within limited computational resources. We characterize the condition under which the multi-objective search algorithm behaves the same with and without surrogate modeling, thus showing the accuracy of our approach. We evaluate our approach by applying it to an industrial ADAS system. Our experiment shows that our approach automatically identifies test cases indicating critical ADAS behaviors. Further, we show that combining our search algorithm with surrogate modeling improves the quality of the generated test cases, especially under tight and realistic computational resources.

Improving fault localization for Simulink models using search-based testing and prediction models

“…While our approach is not particularly tied to any modeling or programming language, we apply our work to Simulink since, in some domains (e.g., automotive), it is expensive to execute Simulink models and to characterize their expected behaviour [4], [19]. This is because Simulink models include computationally expensive physical models [20], and their outputs are complex continuous signals [19]. We identify three alternative test objectives that aim to generate test cases exercising diverse parts of the underlying code and adapt these objectives to Simulink models [15], [16], [21].…”

Testing vision-based control systems using learnable evolutionary algorithms

“…We cast the problem of computing ADAS critical test scenarios as a multi-objective search optimization problem [23] where the ADAS outputs specifying its critical behaviors act as the search fitness functions. We use the Non-dominated Sorting Genetic Algorithm version 2 (NS-GAII) [15,23], which has been previously applied to several software engineering problems including ADAS testing [5]. The NSGAII algorithm generates a set of solutions forming a Pareto nondominated front [15,23].…”

“…Search-based system testing has been previously used to automate test generation for ADAS [11]. For example, it has been applied to a vehicle-to-vehicle braking assistance [9], an autonomous parking [10] and a pedestrian detection system [5]. Bühler and Wegener [9,10] base their testing on a single-objective search algorithm, while Ben Abdessalem et.…”

“…al. [5] use a multi-objective search algorithm. In contrast to our work, none of these approaches consider (static) environment variables in the test input space, and they vary only mobile objects' variables in test scenarios.…”

“…Similar to existing work [5,9,10,29], we rely on evolutionary search techniques [23] to help engineers explore the complex input space of ADAS and to identify critical test scenarios. It is argued that for testing at the system level, search-based techniques are best suited [37].…”