Self-Motivated Multi-Agent Exploration

Abstract

In cooperative multi-agent reinforcement learning (CMARL), it is critical for agents to achieve a balance between self-exploration and team collaboration. However, agents can hardly accomplish the team task without coordination and they would be trapped in a local optimum where easy cooperation is accessed without enough individual exploration. Recent works mainly concentrate on agents' coordinated exploration, which brings about the exponentially grown exploration of the state space. To address this issue, we propose Self-Motivated Multi-Agent Exploration (SMMAE), which aims to achieve success in team tasks by adaptively finding a trade-off between self-exploration and team cooperation. In SM-MAE, we train an independent exploration policy for each agent to maximize their own visited state space. Each agent learns an adjustable exploration probability based on the stability of the joint team policy. The experiments on highly cooperative tasks in Star-Craft II micromanagement benchmark (SMAC) demonstrate that SMMAE can explore task-related states more efficiently, accomplish coordinated behaviours and boost the learning performance.

Intrinsic Action Tendency Consistency for Cooperative Multi-Agent Reinforcement Learning

“…There exist two major problems: how to design an intrinsic reward to guide agents' unified action tendency and how to integrate the intrinsic rewards into the CTDE framework? In MARL, there are plenty of works designing intrinsic rewards including curiosity-based incentives (Böhmer, Rashid, and Whiteson 2019;Hernandez-Leal, Kartal, and Taylor 2019;Iqbal and Sha 2019;Zhang et al 2023), the mutual influence among agents (Chitnis et al 2020;Jaques et al 2019;Wang et al 2019) and other specific designs (Strouse et al 2018;Ma et al 2022;Mguni et al 2021;Du et al 2019). However, most of them are designed to enhance exploration and employed in independent training ways, which suffer from unstable dynamics of environments.…”