Reinforcement learning in feedback control

Hafner, Roland; Riedmiller, Martin

doi:10.1007/s10994-011-5235-x

Cited by 190 publications

(106 citation statements)

References 31 publications

Supporting

Mentioning

105

Contrasting

Unclassified

Order By: Relevance

“…The Augmented random search (ARS) algorithm relies on the augmentations of the basic version of the random search algorithm that is in turn built on accomplished and proven heuristics which is utilized in deep reinforcement learning strategies and techniques [28][29][30][31]. The prime problem of policy search based on augmentation can be formulated and analyzed as being a continuous problem of searching i.e a continuous search problem.…”

Section: Augmented Random Search Algorithm For Quadcoptermentioning

confidence: 99%

Augmented Random Search for Quadcopter Control: An alternative to Reinforcement Learning

Tiwari¹,

Nadimpalli²

2019

IJITCS

View full text Add to dashboard Cite

Model-based reinforcement learning strategies are believed to exhibit more significant sample complexity than model-free strategies to control dynamical systems, such as quadcopters. This belief that Model-based strategies that involve the use of welltrained neural networks for making such high-level decisions always give better performance can be dispelled by making use of Model-free policy search methods. This paper proposes the use of a model-free random searching strategy, called Augmented Random Search (ARS), which is a better and faster approach of linear policy training for continuous control tasks like controlling a Quadcopter's flight. The method achieves state-of-the-art accuracy by eliminating the use of too much data for the training of neural networks that are present in the previous approaches to the task of Quadcopter control. The paper also highlights the performance results of the searching strategy used for this task in a strategically designed task environment with the help of simulations. Reward collection performance over 1000 episodes and agent's behavior in flight for augmented random search is compared with that of the behavior for reinforcement learning state-of-the-art algorithm, called Deep Deterministic policy gradient(DDPG) Our simulations and results manifest that a high variability in performance is observed in commonly used strategies for sample efficiency of such tasks but the built policy network of ARS-Quad can react relatively accurately to step response providing a better performing alternative to reinforcement learning strategies.

show abstract

Section: Augmented Random Search Algorithm For Quadcoptermentioning

confidence: 99%

Augmented Random Search for Quadcopter Control: An alternative to Reinforcement Learning

Tiwari¹,

Nadimpalli²

2019

IJITCS

View full text Add to dashboard Cite

show abstract

“…Previous work used the derivative of the value-function estimate, which is not guaranteed to have compatible function approximation, and can lead to problems when the value-function is estimated using functions such as rectifiers that are not smooth (Prokhorov and Wunsch, 1997;Hafner and Riedmiller, 2011;Heess et al, 2015;Lillicrap et al, 2015).…”

Section: Deviator-actor-critic (Dac) Modelmentioning

confidence: 99%

Grammars for Games: A Gradient-Based, Game-Theoretic Framework for Optimization in Deep Learning

Balduzzi

2016

Front. Robot. AI

View full text Add to dashboard Cite

Deep learning is currently the subject of intensive study. However, fundamental concepts such as representations are not formally defined -researchers "know them when they see them" -and there is no common language for describing and analyzing algorithms. This essay proposes an abstract framework that identifies the essential features of current practice and may provide a foundation for future developments. The backbone of almost all deep learning algorithms is backpropagation, which is simply a gradient computation distributed over a neural network. The main ingredients of the framework are, thus, unsurprisingly: (i) game theory, to formalize distributed optimization; and (ii) communication protocols, to track the flow of zeroth and first-order information. The framework allows natural definitions of semantics (as the meaning encoded in functions), representations (as functions whose semantics is chosen to optimized a criterion), and grammars (as communication protocols equipped with first-order convergence guarantees). Much of the essay is spent discussing examples taken from the literature. The ultimate aim is to develop a graphical language for describing the structure of deep learning algorithms that backgrounds the details of the optimization procedure and foregrounds how the components interact. Inspiration is taken from probabilistic graphical models and factor graphs, which capture the essential structural features of multivariate distributions.

show abstract

“…MLPs output a smooth approximation of training target data. With this in mind, throughout the learning tasks we will apply a smooth reward function, proposed in [10], that can generate state-action value functions easier to approximate by type of function approximator. Having defined a target point s target , a region of width δ is defined around the target where the cost decreases smoothly, from 95% of a base cost down to 0.…”

Section: Learning Tasksmentioning

confidence: 99%

Learning robotic soccer controllers with the Q-Batch update-rule

Cunha

Serra

Lau

et al. 2014

2014 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC)

View full text Add to dashboard Cite

Robotic soccer provides a rich environment for the development of Reinforcement Learning controllers. The competitive environment imposes strong requirements on performance of the developed controllers. RL offers a valuable alternative for the development of efficient controllers while avoiding the hassle of parameter tuning a hand coded policy. This paper presents the application of a recently proposed Batch RL updaterule to learn robotic soccer controllers in the context of the RoboCup Middle Size League. The Q-Batch update-rule exploits the episodic structure of the data collection phase of Batch RL to efficiently evaluate and improve the learned policy. Three different learning tasks, with increasing difficulty, were developed and applied on a simulated environment and later on the physical robot. The performance of the learned controllers is mostly compared to hand-tuned controllers while some comparisons with other RL methods were performed. Results show that the proposed approach is able to learn the tasks in a reduced amount of time, even outperforming existing hand-coded solutions.

show abstract

Reinforcement learning in feedback control

Cited by 190 publications

References 31 publications

Augmented Random Search for Quadcopter Control: An alternative to Reinforcement Learning

Augmented Random Search for Quadcopter Control: An alternative to Reinforcement Learning

Grammars for Games: A Gradient-Based, Game-Theoretic Framework for Optimization in Deep Learning

Learning robotic soccer controllers with the Q-Batch update-rule

Contact Info

Product

Resources

About