Attention-Based Fault-Tolerant Approach for Multi-Agent Reinforcement Learning Systems

Abstract

The aim of multi-agent reinforcement learning systems is to provide interacting agents with the ability to collaboratively learn and adapt to the behavior of other agents. Typically, an agent receives its private observations providing a partial view of the true state of the environment. However, in realistic settings, the harsh environment might cause one or more agents to show arbitrarily faulty or malicious behavior, which may suffice to allow the current coordination mechanisms fail. In this paper, we study a practical scenario of multi-agent reinforcement learning systems considering the security issues in the presence of agents with arbitrarily faulty or malicious behavior. The previous state-of-the-art work that coped with extremely noisy environments was designed on the basis that the noise intensity in the environment was known in advance. However, when the noise intensity changes, the existing method has to adjust the configuration of the model to learn in new environments, which limits the practical applications. To overcome these difficulties, we present an Attention-based Fault-Tolerant (FT-Attn) model, which can select not only correct, but also relevant information for each agent at every time step in noisy environments. The multihead attention mechanism enables the agents to learn effective communication policies through experience concurrent with the action policies. Empirical results showed that FT-Attn beats previous state-of-the-art methods in some extremely noisy environments in both cooperative and competitive scenarios, much closer to the upper-bound performance. Furthermore, FT-Attn maintains a more general fault tolerance ability and does not rely on the prior knowledge about the noise intensity of the environment.

Deep multiagent reinforcement learning: challenges and directions

“…Others use metrics to identify relevant messages, such as attention mechanisms. In its simplest form, this is a vector of importance weights (Peng et al 2018;Gu et al 2021;Mao et al 2020). An alternative is to keep confidence scores about states (Da Silva et al 2017).…”

Comparative Analysis of Routing Schemes Based on Machine Learning

“…In order to ensure the policy consistency of this multiagent cooperative routing model, Ref. [50] used the latest multiagent deep deterministic policy gradient (MADDPG) algorithm [51] to train the model. e final experimental results show that the reinforcement learning intelligent routing algorithm based on the offline link weight has better load balancing characteristics than the shortest path routing, that is, the shorter router average waiting time.…”

Attention-Based Communication and Control for Multi-UAV Path Planning

“…However, existing attention-based CTDE MADRL methods allow inter-agent communication only during the centralized training phase [12]- [14], so can be brittle during the distributed execution phase under dynamic environments. Furthermore, for multi-UAV path planning, these methods consider too many [13] or too few attention heads [12], [14]. By conducting numerical experiments, we found that a single attention head can attend to up to one other UAV agent (e.g., the nearest UAV), which may not be sufficient to avoid collision in a congested area (e.g., near the destination).…”