A Survey on Transformers in Reinforcement Learning

Li, Wenzhe; Luo, Hao; Lin, Zichuan; Zhang, Chongjie; Lu, Zongqing; Ye, Deheng

doi:10.48550/arxiv.2301.03044

Cited by 6 publications

(3 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, the transformer architecture [21] has revolutionized the learning paradigm across numerous fields of artificial intelligence and proven superior performance over the convolutional neural network and RNN. Inspired by the success of transformers in other domains (mainly natural language processing), there has been a surge of interest in applying these scalable, attention-based models to RL [22].…”

Section: Transformers In Reinforcement Learningmentioning

confidence: 99%

Full communication memory networks for team-level cooperation learning

Wang

Sartoretti

2023

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

Communication in multi-agent systems is a key driver of team-level cooperation, for instance allowing individual agents to augment their knowledge about the world in partially-observable environments. In this paper, we propose two reinforcement learning-based multi-agent models, namely FCMNet and FCMTran. The two models both allow agents to simultaneously learn a differentiable communication mechanism that connects all agents as well as a common, cooperative policy conditioned upon received information. FCMNet utilizes multiple directional recurrent neural networks to sequentially transmit and encode the current observation-based messages sent by every other agent at each timestep. FCMTran further relies on the encoder of a modified transformer to simultaneously aggregate multiple self-generated messages sent by all agents at the previous timestep into a single message that is used in the current timestep. Results from evaluating our models on a challenging set of StarCraft II micromanagement tasks with shared rewards show that FCMNet and FCMTran both outperform recent communication-based methods and value decomposition methods in almost all tested StarCraft II micromanagement tasks. We further improve the performance of our models by combining them with value decomposition techniques; there, in particular, we show that FCMTran with value decomposition significantly pushes the state-of-the-art on one of the hardest benchmark tasks without any task-specific tuning. We also investigate the robustness of FCMNet under communication disturbances (i.e., binarized messages, random message loss, and random communication order) in an asymmetric collaborative pathfinding task with individual rewards, demonstrating FMCNet's potential applicability in real-world robotic tasks.

show abstract

Section: Transformers In Reinforcement Learningmentioning

confidence: 99%

Full communication memory networks for team-level cooperation learning

Wang

Sartoretti

2023

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

show abstract

“…Indeed, the limitations of RCSL are not confined to stochastic settings. RCSL methods may exhibit suboptimal performance even in deterministic environments when subopti-mal data is prevalent (Li et al 2023). In deterministic environments, the uncertainty and approximation errors within the behavior policy introduce a form of stochasticity that resembles environmental stochasticity.…”

Section: Stitching In Offline Rlmentioning

confidence: 99%

Medicine-Engineering Interdisciplinary Research Based on Bibliometric Analysis: A Case Study on Medicine-Engineering Institutional Cooperation of Shanghai Jiao Tong University

Wang

Cui

Deng

2022

J. Shanghai Jiaotong Univ. (Sci.)

View full text Add to dashboard Cite

This article aims to provide reference for medicine-engineering interdisciplinary research. Targeted at the scientific literature and patent literature published by Shanghai Jiao Tong University, this article attempts to set up co-occurrence matrix of medicine-engineering institutional information which was extracted from address fields of the papers, so as to construct the medicine-engineering intersection datasets. The dataset of scientific literature was analyzed using bibliometrics and visualization methods from multiple dimensions, and the most active factors, such as trends of output, journal and subject distribution, were identified from the indicators of category normalized citation impact (CNCI), times cited, keywords, citation topics and the degree of medicine-engineering interdisplinary. Research on hotspots and trends was discussed in detail. Analyses of the dataset of patent literature showed research themes and measured the degree for technology convergence of medicine-engineering.

show abstract

“…However, despite the rapid adoption of the transformer architecture in reinforcement learning (RL) after the release of Decision Transformer (DT) (Chen et al, 2021;Hu et al, 2022;Agarwal et al, 2023;Li et al, 2023), models with in-context learning capabilities appeared only recently. This delay is caused by a number of reasons.…”

Section: Introductionmentioning

confidence: 99%