Generalized exploration in policy search

Hoof, Herke van; Tanneberg, Daniel; Peters, Jan

doi:10.1007/s10994-017-5657-1

Cited by 8 publications

(6 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This policy was optimized by a standard RL algorithm that did not account for the dependence of actions. In [11], a policy was analyzed whose parameters were incremented by the OU stochastic process. Essentially, this resulted in autocorrelated random components of actions.…”

Section: A Stochastic Dependence Between Actionsmentioning

confidence: 99%

ACERAC: Efficient Reinforcement Learning in Fine Time Discretization

Wawrzyński

Łyskawa

2024

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

One of the main goals of reinforcement learning (RL) is to provide a way for physical machines to learn optimal behavior instead of being programmed. However, effective control of the machines usually requires fine time discretization. The most common RL methods apply independent random elements to each action, which is not suitable in that setting. It is not feasible because it causes the controlled system to jerk and does not ensure sufficient exploration since a single action is not long enough to create a significant experience that could be translated into policy improvement. In our view, these are the main obstacles that prevent the application of RL in contemporary control systems. To address these pitfalls, in this article, we introduce an RL framework and adequate analytical tools for actions that may be stochastically dependent in subsequent time instances. We also introduce an RL algorithm that approximately optimizes a policy that produces such actions. It applies experience replay (ER) to adjust the likelihood of sequences of previous actions to optimize expected n-step returns that the policy yields. The efficiency of this algorithm is verified against four other RL methods [continuous deep advantage updating (CDAU), proximal policy optimization (PPO), soft actor-critic (SAC), and actor-critic with ER (ACER)] in four simulated learning control problems (Ant, HalfCheetah, Hopper, and Walker2D) in diverse time discretization. The algorithm introduced here outperforms the competitors in most cases considered.

show abstract

Section: A Stochastic Dependence Between Actionsmentioning

confidence: 99%

ACERAC: Efficient Reinforcement Learning in Fine Time Discretization

Wawrzyński

Łyskawa

2024

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…In this experiment, the evolutionary strategies were given more frames, but still ran faster due to better parallelizability. Van Hoof et al (2017) apply auto-correlated noise in parameter space. This noise is distributed similarly to that proposed by Morimoto and Doya (2001) and , but applied to the parameters rather than the actions.…”

Section: Parameter-space Perturbing Strategiesmentioning

confidence: 99%

A Survey of Exploration Methods in Reinforcement Learning

Amin¹,

Gomrokchi²,

Satija³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Exploration is an essential component of reinforcement learning algorithms, where agents need to learn how to predict and control unknown and often stochastic environments. Reinforcement learning agents depend crucially on exploration to obtain informative data for the learning process as the lack of enough information could hinder effective learning. In this article, we provide a survey of modern exploration methods in (Sequential) reinforcement learning, as well as a taxonomy of exploration methods.

show abstract

“…In [5] a policy was analyzed whose parameters were incremented by the autoregressive stochastic process. Essentially, this resulted in autocorrelated random components of actions.…”

Section: Stochastic Dependence Between Actionsmentioning

confidence: 99%

“…This section presents hyperparameters used in simulations reported in Sec. 5. All algorithms used the discount factor equal to 0.99.…”

Section: A Algorithms' Hyperparametersmentioning

confidence: 99%

A framework for reinforcement learning with autocorrelated actions

Szulc¹,

Łyskawa²,

Wawrzyński³

2020

Preprint

View full text Add to dashboard Cite

The subject of this paper is reinforcement learning. Policies are considered here that produce actions based on states and random elements autocorrelated in subsequent time instants. Consequently, an agent learns from experiments that are distributed over time and potentially give better clues to policy improvement. Also, physical implementation of such policies, e.g. in robotics, is less problematic, as it avoids making robots shake. This is in opposition to most RL algorithms which add white noise to control causing unwanted shaking of the robots. An algorithm is introduced here that approximately optimizes the aforementioned policy. Its efficiency is verified for four simulated learning control problems (Ant, HalfCheetah, Hopper, and Walker2D) against three other methods (PPO, SAC, ACER). The algorithm outperforms others in three of these problems.

show abstract

Generalized exploration in policy search

Cited by 8 publications

References 26 publications

ACERAC: Efficient Reinforcement Learning in Fine Time Discretization

ACERAC: Efficient Reinforcement Learning in Fine Time Discretization

A Survey of Exploration Methods in Reinforcement Learning

A framework for reinforcement learning with autocorrelated actions

Contact Info

Product

Resources

About