Stratified Experience Replay: Correcting Multiplicity Bias in Off-Policy Reinforcement Learning

Daley, Brett; Hickert, Cameron; Amato, Christopher

doi:10.48550/arxiv.2102.11319

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experience Replay (Lin, 1992) was a fundamental idea to reinforcement learning and is still being investigated by many researchers to understand its contributions and propose improvements (Andrychowicz et al, 2017;Pan et al, 2018;Novati and Koumoutsakos, 2019;Zha et al, 2019;Fedus et al, 2020;Wei et al, 2021;Daley et al, 2021). In this work, we present a method that addresses the experience memory and how we can model and explore it to make the agents with ER more efficient in using smaller amounts of data.…”

Section: Literature Review and Related Workmentioning

confidence: 99%

Improving Experience Replay through Modeling of Similar Transitions' Sets

Neves¹,

Batisteli²,

Lopes³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this work, we propose and evaluate a new reinforcement learning method, COMPact Experience Replay (COMPER), which uses temporal difference learning with predicted target values based on recurrence over sets of similar transitions, and a new approach for experience replay based on two transitions memories. Our objective is to reduce the required number of experiences to agent training regarding the total accumulated rewarding in the long run. Its relevance to reinforcement learning is related to the small number of observations that it needs to achieve results similar to that obtained by relevant methods in the literature, that generally demand millions of video frames to train an agent on the Atari 2600 games. We report detailed results from five training trials of COMPER for just 100,000 frames and about 25,000 iterations with a small experiences memory on eight challenging games of Arcade Learning Environment (ALE). We also present results for a DQN agent with the same experimental protocol on the same games set as the baseline.To verify the performance of COMPER on approximating a good policy from a smaller number of observations, we also compare its results with that obtained from millions of frames presented on the benchmark of ALE.

show abstract

Section: Literature Review and Related Workmentioning

confidence: 99%