Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction

Abstract

Multiagent reinforcement learning (MARL) is commonly considered to suffer from non-stationary environments and exponentially increasing policy space. It would be even more challenging when rewards are sparse and delayed over long trajectories. In this paper, we study hierarchical deep MARL in cooperative multiagent problems with sparse and delayed reward. With temporal abstraction, we decompose the problem into a hierarchy of different time scales and investigate how agents can learn high-level coordination based on the independent skills learned at the low level. Three hierarchical deep MARL architectures are proposed to learn hierarchical policies under different MARL paradigms. Besides, we propose a new experience replay mechanism to alleviate the issue of the sparse transitions at the high level of abstraction and the non-stationarity of multiagent learning. We empirically demonstrate the effectiveness of our approaches in two domains with extremely sparse feedback: (1) a variety of Multiagent Trash Collection tasks, and (2) a challenging online mobile game, i.e., Fever Basketball Defense.Most previous works learn cooperative polices directly over primitive action spaces and usually perform well in environments with dense reward. However, in many real-world scenarios rewards are Preprint. Under review.

HAMMER: Multi-Level Coordination of Reinforcement Learning Agents via Learned Messaging

“…The MAXQ algorithm was designed to provide for a hierarchical break-down of the reinforcement learning problem by decomposing the value function for the main problem into a set of value functions for the sub-problems [5]. Tang et al [51] used temporal abstraction to let agents learn high-level coordination and independent skills at different temporal scales together. Kumar et al [16] presented another framework benefiting from temporal abstractions to achieve coordination among agents with reduced communication complexities.…”

A Review of Cooperative Multi-Agent Deep Reinforcement Learning

“…In [140] a cooperative problem with sparse and delayed rewards is considered, in which each agent accesses a local observation, takes a local action, and submit the joint action into the environment to get the local rewards. Each agent has some low-level and high-level actions to take such that the problem of the task selection for each agent can be modeled as a hierarchical RL problem.…”

Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning

“…It basically decomposes a multiagent problem into a collection of simultaneous single-agent problems that share the same environment. Even when this approach does not address the non-stationarity problem introduced by the changing policies of the other agents, it nonetheless commonly serves as a strong benchmark for a range of MAS [31,33,27].…”