Self-Attentional Credit Assignment for Transfer in Reinforcement Learning

Ferret, Johan; Marinier, Raphaël; Geist, Matthieu; Pietquin, Olivier

doi:10.24963/ijcai.2020/368

Cited by 10 publications

(7 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Observations contain the position and velocity of the arm and the gripper, as well as the target position, S = R 13 . Again, we use the dense binary reward signal defined in Eq.…”

Section: Environmentsmentioning

confidence: 99%

Discriminative Reward Co-Training

Altmann,

Ritz,

Zorn

et al. 2023

Preprint

View full text Add to dashboard Cite

We propose discriminative reward co-training (DIRECT) as an extension to deep reinforcement learning algorithms. Building upon the concept of self-imitation learning (SIL), we introduce an imitation buffer to store beneficial trajectories generated by the policy, determined by their return. A discriminator network is trained concurrently to the policy to distinguish between trajectories generated by the current policy and beneficial trajectories generated by previous policies. The discriminator’s verdict is used to construct a reward signal for optimizing the policy. By interpolating prior experience, DIRECT is able to act as a reward surrogate, steering policy optimization towards more valuable regions of the reward landscape, thus, towards learning an optimal policy. In this article we formally introduce the additional components, their intended purpose and parameterization, and define a unified training procedure. To reveal insights into the mechanics of the proposed architecture, we provide evaluations of the introduced hyperparameters. Further benchmark evaluations in various discrete and continuous control environments provide evidence that DIRECT is especially beneficial in environments possessing sparse rewards, hard exploration tasks, and shifting circumstances. Our results show that DIRECT outperforms state-of-the-art algorithms in those challenging scenarios by providing a surrogate reward to the policy and directing the optimization towards valuable areas.

show abstract

“…Observations contain the position and velocity of the arm and the gripper, as well as the target position, S = R 13 . Again, we use the dense binary reward signal defined in Eq.…”

Section: Environmentsmentioning

confidence: 99%

Discriminative Reward Co-Training

Altmann,

Ritz,

Zorn

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Transformers have also seen success at the reward redistribution problem. An alternative model to RUDDER was introduced as SECRET [9]. Rather than using LSTMs, SECRET's reward redistribution model is instead realized as a transformer decoder network [34] which learns to predict the sign of 𝑟 𝑡 at every timestep.…”

Section: Transformermentioning

confidence: 99%

“…However, reward shaping is often difficult because it requires environment-specific knowledge. For single agent problems, frameworks like RUDDER and SECRET [1,9] allow constructing neural network models for reward redistribution; learning how sparse delayed rewards can be transformed into dense rewards for effective policy optimization. Unfortunately, in the multi-agent reinforcement learning (MARL) setting, sparse and delayed rewards have not been explored extensively [13].…”

Section: Introductionmentioning

confidence: 99%

Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning

She¹,

Gupta²,

Kochenderfer³

2022

Preprint

View full text Add to dashboard Cite

Sparse and delayed rewards pose a challenge to single agent reinforcement learning. This challenge is amplified in multi-agent reinforcement learning (MARL) where credit assignment of these rewards needs to happen not only across time, but also across agents. We propose Agent-Time Attention (ATA), a neural network model with auxiliary losses for redistributing sparse and delayed rewards in collaborative MARL. We provide a simple example that demonstrates how providing agents with their own local redistributed rewards and shared global redistributed rewards motivate different policies. We extend several MiniGrid environments, specifically MultiRoom and DoorKey, to the multi-agent sparse delayed rewards setting. We demonstrate that ATA outperforms various baselines on many instances of these environments. Source code of the experiments is available at https://github.com/jshe/agent-time-attention.

show abstract

“…task success or high returns). Existing approaches complement or modify RL algorithms by either decomposing observed returns as the sum of redistributed rewards along observed trajectories [Arjona-Medina et al, 2019, Ferret et al, 2019, Hung et al, 2019, Raposo et al, 2021 or incorporating hindsight information into the RL process [Harutyunyan et al, 2019, Ferret et al, 2021, Mesnard et al, 2021. Our approach is related but differs in several points: it is tied to (and aims at making sense of) performance improvements instead of outcomes, and it comes from an abstraction over MDPs (which is non-parametric).…”

Section: Related Workmentioning

confidence: 99%

Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Act

Jacq¹,

Ferret²,

Pietquin³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Traditionally, Reinforcement Learning (RL) aims at deciding how to act optimally for an artificial agent. We argue that deciding when to act is equally important. As humans, we drift from default, instinctive or memorized behaviors to focused, thought-out behaviors when required by the situation. To enhance RL agents with this aptitude, we propose to augment the standard Markov Decision Process and make a new mode of action available: being lazy, which defers decision-making to a default policy. In addition, we penalize non-lazy actions in order to encourage minimal effort and have agents focus on critical decisions only. We name the resulting formalism lazy-MDPs. We study the theoretical properties of lazy-MDPs, expressing value functions and characterizing optimal solutions. Then we empirically demonstrate that policies learned in lazy-MDPs generally come with a form of interpretability: by construction, they show us the states where the agent takes control over the default policy. We deem those states and corresponding actions important since they explain the difference in performance between the default and the new, lazy policy. With suboptimal policies as default (pretrained or random), we observe that agents are able to get competitive performance in Atari games while only taking control in a limited subset of states.

show abstract

Self-Attentional Credit Assignment for Transfer in Reinforcement Learning

Cited by 10 publications

References 15 publications

Discriminative Reward Co-Training

Discriminative Reward Co-Training

Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning

Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Act

Contact Info

Product

Resources

About