Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving

Abstract

We examine the problem of adversarial reinforcement learning for multi-agent domains including a rule-based agent. Rule-based algorithms are required in safety-critical applications for them to work properly in a wide range of situations. Hence, every effort is made to find failure scenarios during the development phase. However, as the software becomes complicated, finding failure cases becomes difficult. Especially in multi-agent domains, such as autonomous driving environments, it is much harder to find useful failure scenarios that help us improve the algorithm. We propose a method for efficiently finding failure scenarios; this method trains the adversarial agents using multiagent reinforcement learning such that the tested rule-based agent fails. We demonstrate the effectiveness of our proposed method using a simple environment and autonomous driving simulator. 2 We have an option to define a virtual reward for the player. However, it is often difficult to precisely define the (virtual) reward.

Multi-Agent Vulnerability Discovery for Autonomous Driving with Hazard Arbitration Reward

“…STARS can discover diverse scenarios exhibiting var-ious accident patterns, which is different from previous studies that conduct RL-based adversarial attacks on driving policies [11], [18], [12], [13], [14]. Most previous studies [11], [18], [12], [13] aim to get direct collision scenarios by designing direct collision rewards.…”

“…There exist some studies [11], [12], [13], [14] that adopt the Attack-by-Policy strategy to attack an autonomous driving policy. However, most of these studies aim at directly causing accidents [11], [12], [13], rather than discovering the vulnerability of the under-test policy.…”

“…2) Multi-Attacker Entrypoint: Another relevant study [14] uses multiple adversarial vehicles, and designs a reward to encourage the attackers to reach their own goals as well as to attack the under-test policy. These two parts of reward design are named as the driving reward and adversarial reward, respectively.…”

Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies

“…However, this approach targets a mixed-traffic driving with a single AC and multiple human-driven cars, thus it does not consider complex scenario having more than one non-communicating AC agents. Another work [17] performs adversarial RL for testing a multiagent driving environment by training more than one rulebased driving model. While the results look promising, the approach only covers the cases where the trained adversarial cars are exposed to a single AC.…”

“…There have been attempts to generate adversarial test cases mainly either by using Generative Adversarial Networks (GANs) [14] or optimization based methods [15] [16] targeting the input state of deep neural networks. Furthermore, RL has been used in different styles to test autonomous driving within simulations [17] [18]. However, while RL has shown great results as an adversarial agent, RL is still mainly used for testing ACs in a single-agent environment.…”