QPLEX: Duplex Dueling Multi-Agent Q-Learning

Abstract

We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE requires the consistency of the optimal joint action selection with optimal individual action selections, which is called the IGM (Individual-Global-Max) principle. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may lead to poor policies or even divergence. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), that takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling architecture transforms the IGM principle to easily realized constraints on advantage functions and thus enables efficient value function learning. Theoretical analysis shows that QPLEX solves a rich class of tasks. Empirical experiments on StarCraft II unit micromanagement tasks demonstrate that QPLEX significantly outperforms state-of-the-art baselines in both online and offline task settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional exploration. * Equal contribution.Preprint. Under review.

Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration

“…( 4) Why do we choose to predict target Q ext i for generating intrinsic rewards rather than other choices (Section 5.4)? We will propose several didactic examples and demonstrate the advantage of our method in coordinated exploration, and evaluate our method on the StarCraft II micromanagement (SMAC) benchmark [8] compared with existing state-of-the-art multi-agent reinforcement learning (MARL) algorithms: QPLEX [7], Weighted-QMIX [39], QTRAN [6], QMIX [5], VDN [4], RODE [40], and MAVEN [15].…”

“…The independence of inference module leads to another advantage, that EMC's architecture can be adopted into many value-factorization-based multi-agent algorithms which utilize the CDTE paradigm, i.e., the general function f in Figure 2a can indicate specific (linear, monotonic and IGM) value factorization structures in VDN [4], QMIX [5], and QPLEX [7], respectively. In this paper, we utilize these state-of-the-art algorithms for the inference module.…”

“…With this paradigm, agents' policies are trained with access to global information in a centralized way and executed only based on local histories in a decentralized way. Based on the paradigm of CTDE, many deep MARL methods have been proposed, including VDN [4], QMIX [5], QTRAN [6], and QPLEX [7].…”

“…Despite the current success, since only using simple -greedy exploration strategy, these deep MARL approaches are found ineffective to solve complex coordination tasks that require coordinated and efficient exploration [7]. Exploration has been extensively studied in single-agent reinforcement learning and many advanced methods have been proposed, including pseudo-counts [9,10], curiosity [11,12], and information gain [13].…”

Multiagent Model-based Credit Assignment for Continuous Control

“…This method can apply to competitive settings but does not address the multiagent credit assignment problem. Another line of work using centralised training, decentralised execution, and Q-learning are the implicit credit assignment methods such as [25] [24] [36]. More recently, [43] showed the effectiveness of PPO in multiagent problems for discrete action space domains.…”