Collaborative Evolutionary Reinforcement Learning

Khadka, Shauharda; Majumdar, Somdeb; Nassar, Tarek; Dwiel, Zach; Tumer, Evren; Miret, Santiago; Liu, Yinyin; Tumer, Kagan

doi:10.48550/arxiv.1905.00976

Cited by 9 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These works, while important, do not address the problem of exploring a large state space, and whether this exploration can be improved in multi-agent systems. A recent approach to collaborative evolutionary reinforcement learning [11] shares some similarities with our approach. As in our work, the authors devise a method for learning a population of diverse policies, training using a shared replay buffer among all learners to increase sample efficiency, and dynamically selecting the best learner; however, their work is focused on single-agent tasks and does not incorporate any notion of intrinsic rewards.…”

Section: Related Workmentioning

confidence: 99%

Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning

Iqbal¹,

Sha²

2019

Preprint

View full text Add to dashboard Cite

Sparse rewards are one of the most important challenges in reinforcement learning. In the single-agent setting, these challenges have been addressed by introducing intrinsic rewards that motivate agents to explore unseen regions of their state spaces. Applying these techniques naively to the multi-agent setting results in individual agents exploring independently, without any coordination among themselves. We argue that learning in cooperative multi-agent settings can be accelerated and improved if agents coordinate with respect to what they have explored. In this paper we propose an approach for learning how to dynamically select between different types of intrinsic rewards which consider not just what an individual agent has explored, but all agents, such that the agents can coordinate their exploration and maximize extrinsic returns. Concretely, we formulate the approach as a hierarchical policy where a high-level controller selects among sets of policies trained on different types of intrinsic rewards and the low-level controllers learn the action policies of all agents under these specific rewards. We demonstrate the effectiveness of the proposed approach in a multi-agent learning domain with sparse rewards.

show abstract

Section: Related Workmentioning

confidence: 99%

Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning

Iqbal¹,

Sha²

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Our training algorithm, EGRL, builds on the CERL framework [Khadka et al, 2019] to tackle variable-sized, multi-discrete action settings. Figure 2 illustrates the high level architecture of EGRL.…”

Section: Trainingmentioning

confidence: 99%

“…In addition to the extremely large action space, the reward is end-to-end latency, which is a sparse and noisy learning signal, which we demonstrate is unsuitable for purely gradient-based Deep RL algorithms. Instead, we contribute Evolutionary Graph RL (EGRL), an extension of CERL [Khadka et al, 2019], a population based method which previously performed well in sparse-reward tasks by combining fast policy gradient (PG) learning with a stable evolutionary algorithm (EA). Since the action spaces explored in this paper are several orders of magnitude larger than the ones explored in CERL, we also needed a mechanism to improve the sample-efficiency of the slow EA component.…”

Section: Introductionmentioning

confidence: 99%

Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning

Khadka¹,

Aflalo²,

Marder³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

As modern neural networks have grown to billions of parameters, meeting tight latency budgets has become increasingly challenging. Approaches like compression, sparsification and network pruning have proven effective to tackle this problembut they rely on modifications of the underlying network. In this paper, we look at a complimentary approach of optimizing how tensors are mapped to on-chip memory in an inference accelerator while leaving the network parameters untouched. Since different memory components trade off capacity for bandwidth differently, a sub-optimal mapping can result in high latency. We introduce evolutionary graph reinforcement learning (EGRL) -a method combining graph neural networks, reinforcement learning (RL) and evolutionary search -that aims to find the optimal mapping to minimize latency. Furthermore, a set of fast, stateless policies guide the evolutionary search to improve sample-efficiency. We train and validate our approach directly on the Intel NNP-I chip for inference using a batch size of 1. EGRL outperforms policy-gradient, evolutionary search and dynamic programming baselines on BERT, ResNet-101 and ResNet-50. We achieve 28-78% speed-up compared to the native NNP-I compiler on all three workloads.

show abstract

“…To address this redundancy issue, we apply the cooperative evolutionary reinforcement learning (CERL) approach (Khadka et al 2019). The key idea is to use different hyperparameter settings for each opponent policy, while use an off-policy learning algorithm and a shared experience replay buffer to keep the advantage of concurrently training multiple policies.…”

Section: Marl With Ensemble Trainingmentioning

confidence: 99%

Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games

Shen,

How

2019

Preprint

View full text Add to dashboard Cite

This paper presents an algorithmic framework for learning robust policies in asymmetric imperfect-information games, where the joint reward could depend on the uncertain opponent type (a private information known only to the opponent itself and its ally). In order to maximize the reward, the protagonist agent has to infer the opponent type through agent modeling. We use multiagent reinforcement learning (MARL) to learn opponent models through self-play, which captures the full strategy interaction and reasoning between agents. However, agent policies learned from self-play can suffer from mutual overfitting. Ensemble training methods can be used to improve the robustness of agent policy against different opponents, but it also significantly increases the computational overhead. In order to achieve a good trade-off between the robustness of the learned policy and the computation complexity, we propose to train a separate opponent policy against the protagonist agent for evaluation purposes. The reward achieved by this opponent is a noisy measure of the robustness of the protagonist agent policy due to the intrinsic stochastic nature of a reinforcement learner. To handle this stochasticity, we apply a stochastic optimization scheme to dynamically update the opponent ensemble to optimize an objective function that strikes a balance between robustness and computation complexity. We empirically show that, under the same limited computational budget, the proposed method results in more robust policy learning than standard ensemble training.

show abstract

Collaborative Evolutionary Reinforcement Learning

Cited by 9 publications

References 0 publications

Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning

Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learning

Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning

Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games

Contact Info

Product

Resources

About