ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxation

Abstract

In a multirobot system, a number of cyberphysical attacks (e.g., communication hijack, observation perturbations) can challenge the robustness of agents. This robustness issue worsens in multiagent reinforcement learning because there exists the non-stationarity of the environment caused by simultaneously learning agents whose changing policies affect the transition and reward functions. In this paper, we propose a minimax MARL approach to infer the worst-case policy update of other agents. As the minimax formulation is computationally intractable to solve, we apply the convex relaxation of neural networks to solve the inner minimization problem. Such convex relaxation enables robustness in interacting with peer agents that may have significantly different behaviors and also achieves a certified bound of the original optimization problem. We evaluate our approach on multiple mixed cooperative-competitive tasks and show that our method outperforms the previous state of the art approaches on this topic.

Sparse Adversarial Attack in Multi-agent Reinforcement Learning

“…Adversarial attacks and adversarial training in MARL Lin et al [10], Guo et al [5] attacked one agent in the cMARL environment. Li et al [9], Sun et al [25], Nisioti et al [16] did minimax adversarial training in the cMARL environment, which assumed some agents may behave adversarially against other agents. All these works assumed agent(s) can behave adversarially at any timestep.…”