A Framework for Reinforcement Learning with Autocorrelated Actions

Szulc, Marcin; Łyskawa, Jakub; Wawrzyński, Paweł

doi:10.1007/978-3-030-63833-7_8

Cited by 2 publications

(1 citation statement)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The performance of the ACERAC algorithm is compared using these problems with state-of-the-art RL methods. This article extends [6] in several directions. We introduce here the notion of adjusted noise, which is the input to the noise-value function.…”

mentioning

confidence: 67%

ACERAC: Efficient Reinforcement Learning in Fine Time Discretization

Wawrzyński

Łyskawa

2024

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

One of the main goals of reinforcement learning (RL) is to provide a way for physical machines to learn optimal behavior instead of being programmed. However, effective control of the machines usually requires fine time discretization. The most common RL methods apply independent random elements to each action, which is not suitable in that setting. It is not feasible because it causes the controlled system to jerk and does not ensure sufficient exploration since a single action is not long enough to create a significant experience that could be translated into policy improvement. In our view, these are the main obstacles that prevent the application of RL in contemporary control systems. To address these pitfalls, in this article, we introduce an RL framework and adequate analytical tools for actions that may be stochastically dependent in subsequent time instances. We also introduce an RL algorithm that approximately optimizes a policy that produces such actions. It applies experience replay (ER) to adjust the likelihood of sequences of previous actions to optimize expected n-step returns that the policy yields. The efficiency of this algorithm is verified against four other RL methods [continuous deep advantage updating (CDAU), proximal policy optimization (PPO), soft actor-critic (SAC), and actor-critic with ER (ACER)] in four simulated learning control problems (Ant, HalfCheetah, Hopper, and Walker2D) in diverse time discretization. The algorithm introduced here outperforms the competitors in most cases considered.

show abstract

mentioning

confidence: 67%