Testing of autonomous vehicles using surrogate models and stochastic optimization

Abstract

Advancement in testing and verification methodologies is one of the key requirements for the commercialization and standardization of autonomous driving. Even though great progress has been made, the main challenges encountered during testing of autonomous vehicles, e.g., high number of test scenarios, huge parameter space and long simulation runs, still remain. In order to reduce current testing efforts, we propose an innovative method based on surrogate models in combination with stochastic optimization. The approach presents an iterative zooming-in algorithm aiming to minimize a given cost function and to identify faulty behavior regions within the parameter space. The surrogate model is updated in each iteration and is further used for intensive evaluation tasks, such as exploration and optimization.

Testing machine learning based systems: a systematic mapping

“…The cost of performing MLS testing is particularly challenging, especially in resource-constrained settings (e.g., during system or in-field testing) and in the presence of high dimensional data. Eight papers tackle this problem in the automotive domain (Abdessalem et al 2016(Abdessalem et al , 2018aBeglerovic et al 2017;Zhao and Gao 2018;Bühler and Wegener 2004;Murphy et al 2009;Abeysirigoonawardena et al 2019;Tuncali et al 2018). In this domain, comprehensive in-field testing is prohibitively expensive in terms of required time and resources.…”

“…7 Testing Levels i.e., in principle it may be applicable to any MLS (Aniculaesei et al 2018;Byun et al 2019;Cheng et al 2018a, b;Du et al 2019;Eniser et al 2019;Guo et al 2018;Henriksson et al 2019;Kim et al 2019;Li et al 2018;Ma et al 2018bMa et al , c, d, 2019Murphy et al 2007aMurphy et al , b, 2008Murphy et al , b, 2009Nakajima and Bui 2016, 2019Odena et al 2019;Pei et al 2017;Saha and Kanewala 2019;Sekhon and Fleming 2019;Shen et al 2018;Shi et al 2019;Sun et al 2018a, b;Tian et al 2018;Udeshi and Chattopadhyay 2019;Uesato et al 2019;Xie et al 2018Xie et al , 2019Xie et al , 2011Zhang et al 2018aZhang et al , 2019Zhao and Gao 2018). Around 30% proposed approaches are designed for autonomous systems (Abeysirigoonawardena et al 2019;Beglerovic et al 2017;Bühler and Wegener 2004;Klueck et al 2018;Li et al 2016;Mullins et al 2018;de Oliveira Neves et al 2016;Patel et al 2018;Strickland et al 2018;Wolschke et al 2017;Fremont et al 2019), am...…”

“…Nearly 60% (33 out of 55) of the papers present various methods to generate test inputs for the MLS Fig. 10 Test Artefacts under test (Abdessalem et al 2016(Abdessalem et al , 2018aAbeysirigoonawardena et al 2019;Beglerovic et al 2017;Bühler and Wegener 2004;Du et al 2019;Eniser et al 2019;Fremont et al 2019;Guo et al 2018;Klueck et al 2018;Li et al 2016;Ma et al 2018cMa et al , 2019Majumdar et al 2019;Murphy et al 2007a, b;de Oliveira Neves et al 2016;Odena et al 2019;Pei et al 2017;Sekhon and Fleming 2019;Sun et al 2018a, b;Tian et al 2018;Tuncali et al 2018Tuncali et al , 2019Udeshi et al 2018Udeshi et al , 2019Wolschke et al 2017Wolschke et al , 2018Xie et al 2019;Zhang et al 2018b;Zheng et al 2018b). However, what a test input represents differs across the proposed approaches and highly depends on the domain of the tested system.…”

“…In six papers the generation of inputs using a search-based approach aims to detect collision scenarios for autonomous driving systems. Therefore, their fitness functions use metrics such as distance to other static or dynamic objects (Bühler and Wegener 2004;Abdessalem et al 2016Abdessalem et al , 2018b, time to collision (Tuncali and Fainekos 2019;Beglerovic et al 2017;Abdessalem et al 2016), speed of the vehicle (Tuncali and Fainekos 2019;Abdessalem et al 2018b) or level of confidence in the detection of the object in front of the vehicle (Abdessalem et al 2016(Abdessalem et al , 2018a. In contrast, Mullins et al (2018) aim to identify test inputs for an autonomous system that are located in its performance boundaries, i.e., in the regions of the input space where small alterations to the input can cause transitions in the behaviour, resulting in major performance changes.…”

Optimizing coverage of simulated driving scenarios for the autonomous vehicle

“…Optimization algorithms, machine learning and deep learning models are considered and used. While previous studies have used machine learning techniques to optimize the detection of test cases with higher probability of failures [5], the objective of this paper is to present an algorithm designed to detect a maximum number of failures of the autonomous vehicle command law in the space of input parameters of the simulator. The main industrial restriction, however, is that this goal must be achieved by running as few numerical simulations as possible, in order to minimize the computational power.…”