Efficient online testing for DNN-enabled systems using surrogate-assisted and many-objective optimization

Haq, Fitash Ul; Shin, Donghwan; Briand, Lionel C.

doi:10.1145/3510003.3510188

Cited by 48 publications

(30 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, to evaluate the robustness of the perception module under different weather conditions, researchers may focus more on the configurations of different weathers, such as rain and fog [9,41,51]. To evaluate the safety of an ADS, researchers focus more on the configurations of the trajectories and behaviors of the NPC vehicles [23,28,45].…”

Section: Scenario Descriptionmentioning

confidence: 99%

“…Motivated by these preliminary studies, considering ADS safety testing, we also focus on the configurable attributes for each NPC vehicle, i.e., the route described by the initial and target positions and the trajectory described by a set of waypoints, to search for diverse and critical scenarios. Different from the existing work [23,28] which only mutates the waypoints of each NPC vehicle without specifying the route explicitly, we mutate both the NPC's route and its corresponding waypoints, each of which is formulated by the position (denoted as 𝑝) and velocity (denoted as 𝑣) of the vehicle. In this way, we aim to search for critical scenarios more efficiently.…”

Section: Scenario Descriptionmentioning

confidence: 99%

“…Several ADS testing methods have been proposed to generate critical scenarios efficiently, such as data-driven methods [3,12,35], surrogate models [4,31], and guided searching methods [18,23,28,55]. They usually design risk evaluation metrics (e.g., time to collision and proportion of stopping distance) [29] to guide the generation of critical scenarios.…”

Section: Introductionmentioning

confidence: 99%

“…We evaluated the effectiveness of BehAVExplor on the LGSVL simulator and Baidu Apollo ADS. Specifically, we compared Be-hAVExplor with three baselines, i.e., random testing, AVFuzzer [28] and SAMOTA [23]. The evaluation results show that BehAVExplor discovers more violations (an increase of 61.8% on average) than the baselines (the best one is AVFuzzer).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems

Cheng

Zhou

Xie

2023

Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Testing Autonomous Driving Systems (ADSs) is a critical task for ensuring the reliability and safety of autonomous vehicles. Existing methods mainly focus on searching for safety violations while the diversity of the generated test cases is ignored, which may generate many redundant test cases and failures. Such redundant failures can reduce testing performance and increase failure analysis costs.In this paper, we present a novel behavior-guided fuzzing technique (BehAVExplor) to explore the different behaviors of the ego vehicle (i.e., the vehicle controlled by the ADS under test) and detect diverse violations. Specifically, we design an efficient unsupervised model, called BehaviorMiner, to characterize the behavior of the ego vehicle. BehaviorMiner extracts the temporal features from the given scenarios and performs a clustering-based abstraction to group behaviors with similar features into abstract states. A new test case will be added to the seed corpus if it triggers new behaviors (e.g., cover new abstract states). Due to the potential conflict between the behavior diversity and the general violation feedback, we further propose an energy mechanism to guide the seed selection and the mutation. The energy of a seed quantifies how good it is. We evaluated BehAVExplor on Apollo, an industrial-level ADS, and LGSVL simulation environment. Empirical evaluation results show that BehAVExplor can effectively find more diverse violations than the state-of-the-art. CCS CONCEPTS• Software and its engineering → Software testing and debugging.

show abstract

Section: Scenario Descriptionmentioning

confidence: 99%

Section: Scenario Descriptionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems

Cheng

Zhou

Xie

2023

Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

show abstract

“…To detect safety violations, if any, during the simulation of a complete solution (i.e., the combination of a scenario and an MLC behavior), we measure the distance between the ego vehicle and the vehicle in front for each simulation time step. Based on feedback from domain experts and values provided in [42,46], the value of 1.5 m was chosen for 𝑑 𝑚𝑖𝑛 for the evaluation, which is reasonable as minimum distance behind a stopped car in the front since AVs that have much quicker reaction times than humans [47,48]. If the distance is less then 𝑑 𝑚𝑖𝑛 at any time, the violation is detected and the complete solution is marked as unsafe.…”

Section: Evaluation Subjectsmentioning

confidence: 99%

Identifying the Hazard Boundary of ML-enabled Autonomous Systems Using Cooperative Co-Evolutionary Search

Sharifi¹,

Shin²,

Briand³

et al. 2023

Preprint

View full text Add to dashboard Cite

In Machine Learning (ML)-enabled autonomous systems (MLASs), it is essential to identify the hazard boundary of ML Components (MLCs) in the MLAS under analysis. Given that such boundary captures the conditions in terms of MLC behavior and system context that can lead to hazards, it can then be used to, for example, build a safety monitor that can take any predefined fallback mechanisms at runtime when reaching the hazard boundary. However, determining such hazard boundary for an ML component is challenging. This is due to the space combining system contexts (i.e., scenarios) and MLC behaviors (i.e., inputs and outputs) being far too large for exhaustive exploration and even to handle using conventional metaheuristics, such as genetic algorithms. Additionally, the high computational cost of simulations required to determine any MLAS safety violations makes the problem even more challenging. Furthermore, it is unrealistic to consider a region in the problem space deterministically safe or unsafe due to the uncontrollable parameters in simulations and the non-linear behaviors of ML models (e.g., deep neural networks) in the MLAS under analysis. To address the challenges, we propose MLCSHE (ML Component Safety Hazard Envelope), a novel method based on a Cooperative Co-Evolutionary Algorithm (CCEA), which aims to tackle a high-dimensional problem by decomposing it into two lower-dimensional search subproblems. Moreover, we take a probabilistic view of safe and unsafe regions and define a novel fitness function to measure the distance from the probabilistic hazard boundary and thus drive the search effectively. We evaluate the effectiveness and efficiency of MLCSHE on a complex Autonomous Vehicle (AV) case study. Our evaluation results show that MLCSHE is significantly more effective and efficient compared to a standard genetic algorithm and random search.

show abstract

Improving Search-Based Android Test Generation Using Surrogate Models

Auer

Adler²

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Efficient online testing for DNN-enabled systems using surrogate-assisted and many-objective optimization

Cited by 48 publications

References 35 publications

BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems

BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems

Identifying the Hazard Boundary of ML-enabled Autonomous Systems Using Cooperative Co-Evolutionary Search

Improving Search-Based Android Test Generation Using Surrogate Models

Contact Info

Product

Resources

About