Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Zhao, Rui; Song, Jinming; Hu, Haifeng; Gao, Yang; Wu, Yi; Sun, Zhongqian; Yang, Wei

doi:10.48550/arxiv.2112.11701

Cited by 6 publications

(38 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Training the agents with diverse partners is effective to alleviate over-fitting to specific partners and improve ZSC performance. Population-based training (PBT) methods [26,46,56] have achieved state-of-the-art performance in ZSC. In Fig.…”

Section: Related Workmentioning

confidence: 99%

“…FCP [46] trains a diverse population by setting different random seeds and including partners of level of cooperation skills and architectures. TrajeDi [26] and MEP [56] adopt explicit diversity objective to generate diverse policies as partners and achieve state-of-the-art ZSC performance. Our PECAN also maintains a population of policies.…”

Section: Related Workmentioning

confidence: 99%

“…And humans usually prefer adaptive AI partners [46]. Thus, an adaptive agent with strong ZSC performance is more likely to succeed in human-AI coordination tasks [46,56]. Diversity in RL Diversity is a widely discussed issue in RL.…”

Section: Related Workmentioning

confidence: 99%

“…[9] proposed to use generative model to generate diverse policies. MEP [56] maximizes the population entropy to encourage diversity among a group of policies. EPPO [53] uses mean inner product as diversity enhancement regularization and obtains a mutually distinct policy group.…”

Section: Related Workmentioning

confidence: 99%

“…However, this approach has been shown to result in over-fitting to a single cooperative pattern [19,26] and poor generalization to real human interactions. To address this issue, population-based training (PBT) [26,46,56] has been employed, where a population of diverse partner agents is used to train the ego agent. The variety of behaviors exhibited by these diverse partners can prevent over-fitting to a single cooperative pattern and improve generalization ability of the ego agent when cooperating with real humans.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination

Lou¹,

Guo²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Zero-shot human-AI coordination holds the promise of collaborating with humans without human data. Prevailing methods try to train the ego agent with a population of partners via self-play. However, these methods suffer from two problems: 1) The diversity of a population with finite partners is limited, thereby limiting the capacity of the trained ego agent to collaborate with a novel human; 2) Current methods only provide a common best response for every partner in the population, which may result in poor zero-shot coordination performance with a novel partner or humans. To address these issues, we first propose the policy ensemble method to increase the diversity of partners in the population, and then develop a context-aware method enabling the ego agent to analyze and identify the partner's potential policy primitives so that it can take different actions accordingly. In this way, the ego agent is able to learn more universal cooperative behaviors for collaborating with diverse partners. We conduct experiments on the Overcooked environment, and evaluate the zero-shot human-AI coordination performance of our method with both behavior-cloned human proxies and real humans. The results demonstrate that our method significantly increases the diversity of partners and enables ego agents to learn more diverse behaviors than baselines, thus achieving state-of-theart performance in all scenarios. We also open-source a human-AI coordination study framework on the Overcooked for the convenience of future studies. Codes and demo videos are available at https://sites.google.com/view/pecan-overcooked.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination

Lou¹,

Guo²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Communication-robust multi-agent learning by adaptable auxiliary multi-agent adversary generation

Yuan,

Chen,

Zhang

et al. 2023

Front. Comput. Sci.

View full text Add to dashboard Cite

Communication can promote coordination in cooperative Multi-Agent Reinforcement Learning (MARL). Nowadays, existing works mainly focus on improving the communication efficiency of agents, neglecting that real-world communication is much more challenging as there may exist noise or potential attackers. Thus the robustness of the communication-based policies becomes an emergent and severe issue that needs more exploration. In this paper, we posit that the ego system1) trained with auxiliary adversaries may handle this limitation and propose an adaptable method of Multi-Agent Auxiliary Adversaries Generation for robust Communication, dubbed MA3C, to obtain a robust communication-based policy. In specific, we introduce a novel message-attacking approach that models the learning of the auxiliary attacker as a cooperative problem under a shared goal to minimize the coordination ability of the ego system, with which every information channel may suffer from distinct message attacks. Furthermore, as naive adversarial training may impede the generalization ability of the ego system, we design an attacker population generation approach based on evolutionary learning. Finally, the ego system is paired with an attacker population and then alternatively trained against the continuously evolving attackers to improve its robustness, meaning that both the ego system and the attackers are adaptable. Extensive experiments on multiple benchmarks indicate that our proposed MA3C provides comparable or better robustness and generalization ability than other baselines.

show abstract

Mixture of personality improved spiking actor network for efficient multi-agent cooperation

Ruan

et al. 2023

Front. Neurosci.

View full text Add to dashboard Cite

Adaptive multi-agent cooperation with especially unseen partners is becoming more challenging in multi-agent reinforcement learning (MARL) research, whereby conventional deep-learning-based algorithms suffer from the poor new-player-generalization problem, possibly caused by not considering theory-of-mind theory (ToM). Inspired by the ToM personality in cognitive psychology, where a human can easily resolve this problem by predicting others' intuitive personality first before complex actions, we propose a biologically-plausible algorithm named the mixture of personality (MoP) improved spiking actor network (SAN). The MoP module contains a determinantal point process to simulate the formation and integration of different personality types, and the SAN module contains spiking neurons for efficient reinforcement learning. The experimental results on the benchmark cooperative overcooked task showed that the proposed MoP-SAN algorithm could achieve higher performance for the paradigms with (learning) and without (generalization) unseen partners. Furthermore, ablation experiments highlighted the contribution of MoP in SAN learning, and some visualization analysis explained why the proposed algorithm is superior to some counterpart deep actor networks.

show abstract

Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

Cited by 6 publications

References 24 publications

PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination

PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination

Communication-robust multi-agent learning by adaptable auxiliary multi-agent adversary generation

Mixture of personality improved spiking actor network for efficient multi-agent cooperation

Contact Info

Product

Resources

About