PRIMAL$_2$: Pathfinding Via Reinforcement and Imitation Multi-Agent Learning - Lifelong

Abstract

Multi-agent path finding (MAPF) is an indispensable component of large-scale robot deployments in numerous domains ranging from airport management to warehouse automation. In particular, this work addresses lifelong MAPF (LMAPF) -an online variant of the problem where agents are immediately assigned a new goal upon reaching their current one -in dense and highly structured environments, typical of real-world warehouse operations. Effectively solving LMAPF in such environments requires expensive coordination between agents as well as frequent replanning abilities, a daunting task for existing coupled and decoupled approaches alike. With the purpose of achieving considerable agent coordination without any compromise on reactivity and scalability, we introduce PRIMAL 2 , a distributed reinforcement learning framework for LMAPF where agents learn fully decentralized policies to reactively plan paths online in a partially observable world. We extend our previous work, which was effective in low-density sparsely occupied worlds, to highly structured and constrained worlds by identifying behaviors and conventions which improve implicit agent coordination, and enabling their learning through the construction of a novel local agent observation and various training aids. We present extensive results of PRIMAL 2 in both MAPF and LMAPF environments with up to 1024 agents and compare its performance to complete state-of-the-art planners. We experimentally observe that agents successfully learn to follow ideal conventions and can exhibit selfless coordinated maneuvers that maximize joint rewards. We find that not only does PRIMAL 2 significantly surpass our previous work, it is also able to perform on par and even outperform state-of-theart planners in terms of throughput.

Applications and Challenges of Deep Reinforcement Learning in Multi-robot Path Planning

“…Table 1 summarizes the DRL multi-robot path planning methods and the advantages and limitations of each method. From the information in Table 1, it can be summarized that shared parameter type algorithms such as MADDPG and ME-MADDPG can be used in dynamic and complex environments [1][2][3][4] ; decentralized architectures such as DQN and DDQN can be considered in stable environments [5][6][7] ; large robotic systems facing a large number of dynamic obstacles can be considered using algorithms such as A2C, A3C and TDueling [8][9][10][11] . Validity validated on only a few teams of agents.…”

CTRMs: Learning to Construct Cooperative Timed Roadmaps for Multi-agent Path Planning in Continuous Spaces

“…More recently, some works have attempted to leverage machine-learning techniques for solving MAPF. These techniques learn from planning demonstrations collected offline to directly predict the next actions of agents given the current observations by means of reinforcement learning [9,43,50] or using graph neural networks [38,39]. Despite such progress, it remains challenging to determine how these techniques should be applied to MAPP in continuous spaces due to the inherent limitation that assumes the search space to be given a priori (typically as a grid map).…”

Standby-Based Deadlock Avoidance Method for Multi-Agent Pickup and Delivery Tasks

“…By contrast, some studies [5,8,9,24,25,37] have also considered an application to maze-like environments. For example, Damani et al [5] proposed pathfinding via reinforcement and imitation multiagent learning -lifelong (PRIMAL 2 ), a distributed reinforcement learning framework for a lifelong MAPF (LMAPF), which is a variant of the MAPF in which agents are repeatedly assigned new destinations. However, they assumed that tasks are sparsely generated at random locations, and thus, unlike our environment, no local congestion occurs.…”