LIGS: Learnable Intrinsic-Reward Generation Selection for Multi-Agent Learning

Abstract

Efficient exploration is important for reinforcement learners (RL) to achieve high rewards. In multi-agent systems, coordinated exploration and behaviour is critical for agents to jointly achieve optimal outcomes. In this paper, we introduce a new general framework for improving coordination and performance of multi-agent reinforcement learners (MARL). Our framework, named Learnable Intrinsic-Reward Generation Selection algorithm (LIGS) introduces an adaptive learner, Generator that observes the agents and learns to construct intrinsic rewards online that coordinate the agents' joint exploration and joint behaviour. Using a novel combination of reinforcement learning (RL) and switching controls, LIGS determines the best states to learn to add intrinsic rewards which leads to a highly efficient learning process. LIGS can subdivide complex tasks making them easier to solve and enables systems of RL agents to quickly solve environments with sparse rewards. LIGS can seamlessly adopt existing multi-agent RL algorithms and our theory shows that it ensures convergence to joint policies that deliver higher system performance. We demonstrate the superior performance of the LIGS framework in challenging tasks in Foraging and StarCraft II. To aid coordinated learning, algorithms such as QMIX Rashid et al. (2018), MAVEN Mahajan et al. (2019) and COMA Foerster et al. (2018), so-called centralised critic and decentralised execution (CT-DE) methods use a centralised critic whose role is to estimate the agents' expected returns. The critic * Correspondence to

Intrinsic Action Tendency Consistency for Cooperative Multi-Agent Reinforcement Learning

“…To better be applied in MARL, Some MARL-specific intrinsic reward functions have been proposed, including considering the mutual influence among agents (Chitnis et al 2020;Jaques et al 2019;Wang et al 2019), encouraging agents to reveal or hide their intentions (Strouse et al 2018) and predicting observation with alignment to their neighbors (Ma et al 2022). Besides, Intrinsic rewards without taskoriented bias can increase the diversity of intrinsic reward space, which can be implemented by breaking the extrinsic rewards via credit assignment (Du et al 2019) or using adaptive learners to obtain intrinsic rewards online (Mguni et al 2021). Apart from independent manners to dealing with rewards, EMC (Zheng et al 2021) proposed a curiosity-driven intrinsic reward and introduce an integration way to accomplish the CTDE training paradigm.…”

“…Given the CTDE paradigm, massive deep MARL methods have been proposed including VDN (Sunehag et al 2017), QMIX (Rashid et al 2020), QTRAN (Son et al 2019), QPLEX (Wang et al 2020b) and so forth. Their excellent performance can be attributed to the credit assignments, as rewards are critical as the most direct and fundamental instructional signals to drive behaviors (Silver et al 2021;Zheng et al 2021;Mguni et al 2021).…”

“…There exist two major problems: how to design an intrinsic reward to guide agents' unified action tendency and how to integrate the intrinsic rewards into the CTDE framework? In MARL, there are plenty of works designing intrinsic rewards including curiosity-based incentives (Böhmer, Rashid, and Whiteson 2019;Hernandez-Leal, Kartal, and Taylor 2019;Iqbal and Sha 2019;Zhang et al 2023), the mutual influence among agents (Chitnis et al 2020;Jaques et al 2019;Wang et al 2019) and other specific designs (Strouse et al 2018;Ma et al 2022;Mguni et al 2021;Du et al 2019). However, most of them are designed to enhance exploration and employed in independent training ways, which suffer from unstable dynamics of environments.…”