Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning

Liang, Yongyuan; Li, Bangwei

doi:10.48550/arxiv.2003.13085

Cited by 3 publications

(4 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This lowers the overall exploration cost for the novice by asking for an action to follow in certain states. Other form of advice is Q-values [12,13,29] used to influence the action-selection process of a target agent. Finally, advice can be provided as policy to be followed for a certain number of steps [26] or a batch of RL experience, similar to a demonstration, as in [2,3,7,14,23].…”

Section: Transfer Learning In Rlmentioning

confidence: 99%

“…While that improves advice quality over time, introducing a new super agent results in an additional cost to gather the experiences and to train the underneath model. [4,11,12,19,29] improved the base teacher-student framework by relying on confidence-based and importance-based methods to dynamically select one or more agents as teachers. In these models, an agent can ask for and provide advice simultaneously.…”

Section: Transfer Learning In Rlmentioning

confidence: 99%

See 1 more Smart Citation

Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learning

Castagna,

Dusparic

2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

Transfer learning in Reinforcement Learning (RL) has been widely studied to overcome training challenges in Deep-RL, i.e., exploration cost, data availability and convergence time, by bootstrapping external knowledge to enhance learning phase. While this overcomes the training issues on a novice agent, a good understanding of the task by the expert agent is required for such a transfer to be effective. As an alternative, in this paper we propose Expert-Free Online Transfer Learning (EF-OnTL), an algorithm that enables expert-free real-time dynamic transfer learning in multi-agent system. No dedicated expert agent exists, and transfer source agent and knowledge to be transferred are dynamically selected at each transfer step based on agents’ performance and level of uncertainty. To improve uncertainty estimation, we also propose State Action Reward Next-State Random Network Distillation (sars-RND), an extension of RND that estimates uncertainty from RL agent-environment interaction. We demonstrate EF-OnTL effectiveness against a no-transfer scenario and state-of-the-art advice-based baselines, with and without expert agents, in three benchmark tasks: Cart-Pole, a grid-based Multi-Team Predator-Prey (MT-PP) and Half Field Offense (HFO). Our results show that EF-OnTL achieves overall comparable performance to that of advice-based approaches, while not requiring expert agents, external input, nor threshold tuning. EF-OnTL outperforms no-transfer with an improvement related to the complexity of the task addressed.

show abstract

Section: Transfer Learning In Rlmentioning

confidence: 99%

Section: Transfer Learning In Rlmentioning

confidence: 99%

Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learning

Castagna,

Dusparic

2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

show abstract

“…Learning from others is one essential skill engraved in humans' genes to survive in society. Based on the relationship between teachers and students in human society, a series of research work hopes each agent can learn from others or selectively share its knowledge with others [31][32][33][34]. But it is challenging to specify knowledge in practice, let alone deciding what to share or learn.…”

Section: Related Workmentioning

confidence: 99%

Celebrating Diversity in Shared Multi-Agent Reinforcement Learning

Li¹,

Wu²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, deep multi-agent reinforcement learning (MARL) has shown the promise to solve complex cooperative tasks. Its success is partly because of parameter sharing among agents. However, such sharing may lead agents to behave similarly and limit their coordination capacity. In this paper, we aim to introduce diversity in both optimization and representation of shared multi-agent reinforcement learning. Specifically, we propose an information-theoretical regularization to maximize the mutual information between agents' identities and their trajectories, encouraging extensive exploration and diverse individualized behaviors. In representation, we incorporate agent-specific modules in the shared neural network architecture, which are regularized by L1-norm to promote learning sharing among agents while keeping necessary diversity. Empirical results show that our method achieves stateof-the-art performance on Google Research Football 2 and super hard StarCraft II micromanagement tasks. * Denotes equal contribution.2 Videos available at https://sites.google.com/view/celebrate-diversity-shared Preprint. Under review.

show abstract

“…The common norm that was considered to be the baseline for multi-tasking in the reinforcement learning was based on the approach to follow a transfer-oriented methodology, such as sharing the neural network parameters across related tasks in the environment [52]. Often, this approach met with bottlenecks, such as negative knowledge transfer scenarios, and ambiguity on how to design the reward system for various tasks.…”

Section: Distral (Distill and Transfer Learning)mentioning

confidence: 99%

A Survey of Multi-Task Deep Reinforcement Learning

Varghese

Mahmoud

2020

Electronics

View full text Add to dashboard Cite

Driven by the recent technological advancements within the field of artificial intelligence research, deep learning has emerged as a promising representation learning technique across all of the machine learning classes, especially within the reinforcement learning arena. This new direction has given rise to the evolution of a new technological domain named deep reinforcement learning, which combines the representational learning power of deep learning with existing reinforcement learning methods. Undoubtedly, the inception of deep reinforcement learning has played a vital role in optimizing the performance of reinforcement learning-based intelligent agents with model-free based approaches. Although these methods could improve the performance of agents to a greater extent, they were mainly limited to systems that adopted reinforcement learning algorithms focused on learning a single task. At the same moment, the aforementioned approach was found to be relatively data-inefficient, particularly when reinforcement learning agents needed to interact with more complex and rich data environments. This is primarily due to the limited applicability of deep reinforcement learning algorithms to many scenarios across related tasks from the same environment. The objective of this paper is to survey the research challenges associated with multi-tasking within the deep reinforcement arena and present the state-of-the-art approaches by comparing and contrasting recent solutions, namely DISTRAL (DIStill & TRAnsfer Learning), IMPALA(Importance Weighted Actor-Learner Architecture) and PopArt that aim to address core challenges such as scalability, distraction dilemma, partial observability, catastrophic forgetting and negative knowledge transfer.

show abstract

Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning

Cited by 3 publications

References 13 publications

Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learning

Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learning

Celebrating Diversity in Shared Multi-Agent Reinforcement Learning

A Survey of Multi-Task Deep Reinforcement Learning

Contact Info

Product

Resources

About