Distributed Multi-Agent Deep Reinforcement Learning for Robust Coordination against Noise

MAT-DQN: Toward Interpretable Multi-agent Deep Reinforcement Learning for Coordinated Activities

“…Attention-based method in XRL: In addition to developing research on visual explanations, such as feature-based [7], embedding-based [16], perturbation-based [29], and gradientbased methods [25], incorporation of the attention mechanism in models is one of the most popular methods in XRL [27]. Recently, DA3-X [18] was proposed as an extension of MAT-DQN [17] to demonstrate how decentralized agents build coordination by highlighting the influence of relevant tasks, other agents, and the noise in local observations based on the attention mechanism.…”

“…The reuse of the saliency vector in DA6-X is beneficial not only for performance improvement but also for improved interpretability when compared to DA3-X [18]. As explained earlier, DA6-X agents can flexibly change their strategy of weighing a particular piece of information in their local observation depending on an arbitrary number of available conditional states by reusing the saliency vector.…”

“…Multiactor-attention-critic (MAAC) [6], incorporating the attention mechanism in the style of MADDPG [13], demonstrated the way agents selectively focus on cooperation among themselves using the attention mechanism. [17], [18] developed an approach to establish the interpretability of agents' coordinated behaviors by analyzing the individual agent's attention mechanism and empirically clarified that their agents would selectively identify other agents worthy of coordination. Such findings are expected to improve the explainability of learned behaviors of individuals as well as the efficiency of the entire system.…”

“…To address this challenge, we proposed distributed attentional actor architecture after conditional attention (DA6-X). The model architecture is based on reusing the saliency vector [18] to enhance the interpretability of learned conditional coordinated behaviors in multi-agent DRL (MADRL). DA6-X is a sequential framework that combines a module for recognition of the conditions/situations, with any DRL head (corresponding to 'X'), to output the learned behavior.…”

“…DA6-X is a sequential framework that combines a module for recognition of the conditions/situations, with any DRL head (corresponding to 'X'), to output the learned behavior. Unlike the previous models [17], [18], DA6-X takes twolayered data: conditional states (conditions/situations) and local observations. DA6-X first represents the conditional states using the saliency vector in the former conditional module (CM).…”