Regularized Behavior Value Estimation

Gülçehre, Çağlar; Colmenarejo, Sergio Gómez; Wang, Ziyu; Sygnowski, Jakub; Paine, Thomas; Żołna, Konrad; Chen, Yutian; Hoffman, Matthew; Pascanu, Razvan; Freitas, Nando de

doi:10.48550/arxiv.2103.09575

Cited by 4 publications

(4 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To alleviate the severe extrapolation error in offline agent learning, we plug the filtering mechanism of CRR (Wang et al 2020c) into individual policy learning. This method can implicitly constrain the forward KL divergence between the learning policy and the behavior policy, which is widely used in offline single-agent learning (Wang et al 2020c;Nair et al 2020;Gulcehre et al 2021) and multi-agent learning (Yang et al 2021). Formally, suppose that the type of agent i is J, and its corresponding DPER is denoted as B J .…”

Section: Conservative Policy Learning With Gat-based Criticmentioning

confidence: 99%

Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning

Kuang

Liu

Wang

2023

AAAI

View full text Add to dashboard Cite

Offline multi-agent reinforcement learning (MARL) aims to learn effective multi-agent policies from pre-collected datasets, which is an important step toward the deployment of multi-agent systems in real-world applications. However, in practice, each individual behavior policy that generates multi-agent joint trajectories usually has a different level of how well it performs. e.g., an agent is a random policy while other agents are medium policies. In the cooperative game with global reward, one agent learned by existing offline MARL often inherits this random policy, jeopardizing the utility of the entire team. In this paper, we investigate offline MARL with explicit consideration on the diversity of agent-wise trajectories and propose a novel framework called Shared Individual Trajectories (SIT) to address this problem. Specifically, an attention-based reward decomposition network assigns the credit to each agent through a differentiable key-value memory mechanism in an offline manner. These decomposed credits are then used to reconstruct the joint offline datasets into prioritized experience replay with individual trajectories, thereafter agents can share their good trajectories and conservatively train their policies with a graph attention network (GAT) based critic. We evaluate our method in both discrete control (i.e., StarCraft II and multi-agent particle environment) and continuous control (i.e., multi-agent mujoco). The results indicate that our method achieves significantly better results in complex and mixed offline multi-agent datasets, especially when the difference of data quality between individual trajectories is large.

show abstract

Section: Conservative Policy Learning With Gat-based Criticmentioning

confidence: 99%

Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning

Kuang

Liu

Wang

2023

AAAI

View full text Add to dashboard Cite

show abstract

“…State-action pairs outside of the dataset are therefore never actually experienced and can receive erroneously optimistic value estimates because of extrapolation errors that are not corrected by an environment feedback. The overestimation of value functions can be encouraged in online reinforcement learning, to a certain extent, as it incentivizes agents to explore and learn by trial and error (Gulcehre et al 2021;Schmidhuber 1991). Moreover, in an online setting, if the agent wrongly assigns a high value to a given action, this action will be chosen, the real return will be experimented, and the value of the action will be corrected through bootstrapping.…”

Section: Framing Offline Rl As Anti-explorationmentioning

confidence: 99%

Offline Reinforcement Learning as Anti-exploration

Rezaeifar

Dadashi

Vieillard

et al. 2022

AAAI

View full text Add to dashboard Cite

Offline Reinforcement Learning (RL) aims at learning an optimal control from a fixed dataset, without interactions with the system. An agent in this setting should avoid selecting actions whose consequences cannot be predicted from the data. This is the converse of exploration in RL, which favors such actions. We thus take inspiration from the literature on bonus-based exploration to design a new offline RL agent. The core idea is to subtract a prediction-based exploration bonus from the reward, instead of adding it for exploration. This allows the policy to stay close to the support of the dataset and practically extends some previous pessimism-based offline RL methods to a deep learning setting with arbitrary bonuses. We also connect this approach to a more common regularization of the learned policy towards the data. Instantiated with a bonus based on the prediction error of a variational autoencoder, we show that our simple agent is competitive with the state of the art on a set of continuous control locomotion and manipulation tasks.

show abstract

“…Some algorithms take a different approach, understanding and solving offline RL problems from the perspective of onpolicy learning. R-BVE (Gulcehre et al, 2021) and Onestep RL (Brandfonbrener et al, 2021) both transform off-policy style offline algorithms (such as CRR (Wang et al, 2020), BCQ (Fujimoto et al, 2019), BRAC (Wu et al, 2019)) into on-policy style. Besides, BPPO (Zhuang et al, 2023) finds the online algorithm PPO (Schulman et al, 2017) can directly solve the offline RL due to its inherent conservatism.…”

Section: Related Workmentioning

confidence: 99%