Multi-Agent Concentrative Coordination with Decentralized Task Representation

Abstract

Value-based multi-agent reinforcement learning (MARL) methods hold the promise of promoting coordination in cooperative settings. Popular MARL methods mainly focus on the scalability or the representational capacity of value functions. Such a learning paradigm can reduce agents' uncertainties and promote coordination. However, they fail to leverage the task structure decomposability, which generally exists in real-world multi-agent systems (MASs), leading to a significant amount of time exploring the optimal policy in complex scenarios. To address this limitation, we propose a novel framework Multi-Agent Concentrative Coordination (MACC) based on task decomposition, with which an agent can implicitly form local groups to reduce the learning space to facilitate coordination. In MACC, agents first learn representations for subtasks from their local information and then implement an attention mechanism to concentrate on the most relevant ones. Thus, agents can pay targeted attention to specific subtasks and improve coordination. Extensive experiments on various complex multi-agent benchmarks demonstrate that MACC achieves remarkable performance compared to existing methods.

Learning to Collaborate by Grouping: A Consensus-Oriented Strategy for Multi-Agent Reinforcement Learning

“…LILAC [9] learns a leader to assign roles. Another line of work such as [38,39,28,13], divides the agents into some groups that carry out similar sub-tasks with a specific policy or value function. In our work, we learn more stable and distinguishable group embeddings and further consider the integration of team-level strategy and individual-level decision.…”

GTOC11: Methods and results from the team of Harbin Institute of Technology

“…B2MAPO framework provides a universal modulized plug-andplay architecture, which can conveniently integrate third-party models with few or even without modification to exploit their merits. "CTDE-based joint policy" module in this layer can be implemented by various MARL methods, such as Q-learning-based methods [30,35,39], policy-based methods [41], and actor-critic methods [44].…”

“…Currently, in the field of multi-agent reinforcement learning (MARL), three main Centralized Training and Decentralized Execution (CTDE) [8,10] paradigms are predominantly followed: multi-agent policy gradient methods [41], value decomposition methods [30,35,39], and actor-critic methods [44]. These approaches typically assume independence among agents' policies and update all agents simultaneously.…”

Efficient Communication via Self-supervised Information Aggregation for Online and Offline Multi-agent Reinforcement Learning

“…For better coordination on further applications, some issues like non-stationarity [47], scalability [4] remain to be solved. To solve the non-stationarity caused by the concurrent learning of multiple policies and scalability as the agent number increases, most recent works on MARL adopt the Centralized Training and Decentralized Execution (CTDE) [28,39] paradigm, which includes both value-based methods [57,52,64,76] and policy gradient methods [16,38,68,75], or other techniques like transformer [69]. Under the CTDE paradigm, however, the coordination ability of the learned policies can be fragile due to the partial observability in the multiagent environment, which is a common challenge in many multi-agent tasks [41].…”