Prioritized Experience Replay based on Multi-armed Bandit

Liu, Ximing; Zhu, Tianqing; Jiang, Cuiqing; Ye, Dayong; Zhao, Fuqing

doi:10.1016/j.eswa.2021.116023

Cited by 17 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Du et al [32] proposes a framework to refresh experiences in the replay buffer regarding current policy. Liu et al [33] propose a dynamic experience replay strategy that combines multiple priority-weighted criteria to measure the importance of experiences. Yang and Peng [34] propose the Meta-learning-based ER (MSER) to deal with the computational complexity and the need for careful hyperparameter adjustments in PER.…”

Section: Literature Review and Related Workmentioning

confidence: 99%

Improving Deep Deterministic Policy Gradient with Compact Experience Replay

Neves,

Ishitani,

Patrocínio

2024

Preprint

View full text Add to dashboard Cite

Experience Replay (ER) improves data efficiency in Deep Reinforcement Learning by allowing the agent to revisit past experiences that could contribute to the current policy learning. A recent method called COMPact Experience Replay (COMPER) seeks to improve ER by reducing the required number of experiences for agent training regarding the total accumulated rewards in the long run. This method can approximate good policies on Atari 2600 games on the Arcade Learning Environment (ALE) from a considerably smaller number of frame observations to achieve similar results obtained by DQN-based methods in the literature, which generally demand millions of video frames. Therefore, we conduct a detailed analysis of its components to verify and understand how they operate and how they could improve data efficiency on the Deep Deterministic Policy Gradient (DDPG) algorithm applied to physical control problems in Mujoco's environments. We present an extension of DDPG, named COMPER-DDPG, which uses components from COMPER and achieves better results than the original method in almost all environments used in the experiments with only 50,000 iterations.

show abstract

Section: Literature Review and Related Workmentioning

confidence: 99%

Improving Deep Deterministic Policy Gradient with Compact Experience Replay

Neves,

Ishitani,

Patrocínio

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Mattar & Daw [3] suggested that the reactivation of sequences of behaviourally-relevant experiences during quiet wakefulness and sleep for which the hippocampus is well known [35] is an expression of this sort of prioritized replay. They thereby explained a wealth of experimental findings on the selection of replay experiences in rodents [32,33] as well as humans [5,36].…”

Section: Gain and Needmentioning

confidence: 99%

Exploring Replay

Antonov

Dayan

2023

Preprint

View full text Add to dashboard Cite

Exploration is vital for animals and artificial agents who face uncertainty about their environments due to initial ignorance or subsequent changes. Their choices need to balance exploitation of the knowledge already acquired, with exploration to resolve uncertainty. However, the exact algorithmic structure of exploratory choices in the brain still remains largely elusive. A venerable idea in reinforcement learning is that agents can plan appropriate exploratory choices offline, during the equivalent of quiet wakefulness or sleep. Although offline processing in humans and other animals, in the form of hippocampal replay and preplay, has recently been the subject of highly successful modelling, existing methods only apply to known environments. Thus, they cannot predict exploratory replay choices during learning and/or behaviour in dynamic environments. Here, we extend the theory of Mattar & Daw to examine the potential role of replay in approximately optimal exploration, deriving testable predictions for the patterns of exploratory replay choices in a paradigmatic spatial navigation task. Our modelling provides a normative interpretation of the available experimental data suggestive of exploratory replay. Furthermore, we highlight the importance of sequence replay, and license a range of new experimental paradigms that should further our understanding of offline processing.

show abstract

“…Instead of approaching the sampling strategy, Du et al (2022) proposed a framework to refresh experiences by moving the agent back to past states, executing sequences of actions following its current policy, and storing and reusing new experiences from this process if it turned out better than what the agent previously experienced. (Liu et al, 2022) proposed a dynamic experience replay strategy based on Multarmed Bandit, which combines multiple priority weighted criteria to measure the importance of the experiences and adjust its weights from one episode to another. Yang and Peng (2021) proposed the Meta-learning-based Experience Replay (MSER) applied in DDPG to deal with the computational complexity in PER and its need for careful hyperparameter adjustments.…”

Section: Research On Experience Replay and Some Directions For Future...mentioning

confidence: 99%

Advances and Challenges in Learning from Experience Replay

Neves

Patrocínio

Ishitani

2023

Preprint

View full text Add to dashboard Cite

From the first theoretical propositions in the ’50s, inspired by the neu-roscience and psychology studies about the learning processes in humanbeings and animals, to its application in real-world problems on learn-ing to action, Reinforcement Learning (RL) is still being a fascinating,rich, and complex class of machine learning algorithms. In particular,we will start reviewing its fundamental principles and develop a discus-sion about how a technique called Experience Replay (ER) has beenof fundamental importance in making a variety of methods in most ofthe relevant problems and different domains more data efficient, usingagent experiences to improve its performance. We present some of themore relevant methods in the literature, which base most recent researchon improving RL with ER. Finally, we bring from the recent literaturesome of the main trends, challenges, and advances focused on reviewingand discussing ER formal basement and how to improve its propositionsto make it even more efficient in different methods and domains.

show abstract

Prioritized Experience Replay based on Multi-armed Bandit

Cited by 17 publications

References 12 publications

Improving Deep Deterministic Policy Gradient with Compact Experience Replay

Improving Deep Deterministic Policy Gradient with Compact Experience Replay

Exploring Replay

Advances and Challenges in Learning from Experience Replay

Contact Info

Product

Resources

About