From Few to More: Large-Scale Dynamic Multiagent Curriculum Learning

Abstract

A lot of efforts have been devoted to investigating how agents can learn effectively and achieve coordination in multiagent systems. However, it is still challenging in large-scale multiagent settings due to the complex dynamics between the environment and agents and the explosion of state-action space. In this paper, we design a novel Dynamic Multiagent Curriculum Learning (DyMA-CL) to solve large-scale problems by starting from learning on a multiagent scenario with a small size and progressively increasing the number of agents. We propose three transfer mechanisms across curricula to accelerate the learning process. Moreover, due to the fact that the state dimension varies across curricula, and existing network structures cannot be applied in such a transfer setting since their network input sizes are fixed. Therefore, we design a novel network structure called Dynamic Agent-number Network (DyAN) to handle the dynamic size of the network input. Experimental results show that DyMA-CL using DyAN greatly improves the performance of large-scale multiagent learning compared with state-of-the-art deep reinforcement learning approaches. We also investigate the influence of three transfer mechanisms across curricula through extensive simulations.

Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment

“…Reusing replay buffer and policy distillation are the prevalent auxiliary training methods. [29] improves the efficiency of value-based MATL by reusing the transition data generated in previous scenarios. Inspired by policy distillation [22], Liu et al [16] proposes to transfer the knowledge learned in a single agent to multiple agents and uses the n-step return to approximate the difference of the local environment dynamics.…”

“…Existing CTDE research covers important topics such as division of agents [27], diversification [32] and exploration [19]. Recent works [29,11,1,17,16] have also started to make progress in transfer learning in cooperative MARL. For example, Liu et al [16] use policy distillation [22] to achieve fixed agent transfer learning.…”

“…However, the agent population varies in different tasks in most cases. To solve this problem, DyAN [29] uses a graph neural network to adapt to dynamic agent population. UPDeT [11] uses Transformer [24] to realize a universal and transferable agent policy network to achieve agent-level knowledge transfer.…”

Transformer Based Multi-Agent Framework

“…However, current methods have poor representation learning ability and fail to exploit the common structure underlying the tasks this is because they tend to treat observation from different entities in the environment as an integral part of the whole. Worse yet, conventional models require the input and the output dimensions to be fixed ( [10], [11]), which makes zero-shot transfer impossible. Thus, the application of current methods is limited in real-world applications.…”

Learning Efficient Multi-Agent Cooperative Visual Exploration

“…Algorithm 1 MAPPO Initialize θ, the parameters for policy π and φ, the parameters for critic V , using Orthogonal initialization (Hu et al, 2020) Set learning rate α while step ≤ step max do set data buffer D = {} for i = 1 to num_rollouts do τ = [] empty list for t = 1 to T do for all agents a do p We use the neural SLAM module and the local policy and directly use the trained model provided in origin ANS paper (Chaplot et al, 2020a).…”