Falsification-Based Robust Adversarial Reinforcement Learning

Abstract

Reinforcement learning (RL) has achieved tremendous progress in solving various sequential decision-making problems, e.g., control tasks in robotics. However, RL methods often fail to generalize to safety-critical scenarios since policies are overfitted to training environments. Previously, robust adversarial reinforcement learning (RARL) was proposed to train an adversarial network that applies disturbances to a system, which improves robustness in test scenarios. A drawback of neural-network-based adversaries is that integrating system requirements without handcrafting sophisticated reward signals is difficult. Safety falsification methods allow one to find a set of initial conditions as well as an input sequence, such that the system violates a given property formulated in temporal logic. In this paper, we propose falsification-based RARL (FRARL), the first generic framework for integrating temporal-logic falsification in adversarial learning to improve policy robustness. By using our falsification method, we do not need to construct an extra reward function for the adversary. We evaluate our approach on a braking assistance system and an adaptive cruise control system of autonomous vehicles. Experiments show that policies trained with a falsificationbased adversary generalize better and show less violation of the safety specification in test scenarios than the ones trained without an adversary or with an adversarial network.

Synthesis-guided Adversarial Scenario Generation for Gray-box Feedback Control Systems with Sensing Imperfections

“…These adversarial scenarios are valuable for understanding the shortcomings of the controller at early design stages, which may be hard to expose by random simulations. Moreover, once found, these adversarial scenarios can be used to improve the design, e.g., see [19], [21], [43].…”

Transfer Learning for Efficient Iterative Safety Validation

“…Uesato et al (2019) use previous versions of a system to train a failure classifier that predicts which initial conditions of a system will lead to failure, but their approach is not applicable to sequential decision making problems of the type we consider. Wang, Nair, and Althoff (2020) alternately train an agent and perform safety validation on it to improve robustness. On each iteration, the safety validation algorithm starts with the parameters from the previous iteration to improve efficiency.…”