A comparison of learning speed and ability to cope without exploration between DHP and TD(0)

Fairbank, Michael; Alonso, Eduardo

doi:10.1109/ijcnn.2012.6252569

Cited by 8 publications

(8 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An explicit comparison in learning speed between these two different methods shows the speed up in learning can be of several orders of magnitude when VGL methods are used (Fairbank & Alonso, 2012c), and this confirms the automatic exploration and trajectory bending of VGL.…”

Section: Proof Of Optimalitymentioning

confidence: 64%

Back to optimality: a formal framework to express the dynamics of learning optimal behavior

2015

Self Cite

View full text Add to dashboard Cite

Citation: Alonso, E., Fairbank, M. & Mondragon, E. (2015). Back to optimality: a formal framework to express the dynamics of learning optimal behavior. Adaptive Behavior, 23(4), pp. 206-215. doi: 10.1177/1059712315589355 This is the accepted version of the paper.This version of the publication may differ from the final published version. Greetings, and thank you for publishing with SAGE. We have prepared this page proof for your review. Please respond to each of the below queries by digitally marking this PDF using Adobe Reader. PermanentClick "Comment" in the upper right corner of Adobe Reader to access the mark-up tools as follows:For textual edits, please use the "Annotations" tools. Please refrain from using the two tools crossed out below, as data loss can occur when using these tools.For formatting requests, questions, or other complicated changes, please insert a comment using "Drawing Markups." Abstract Whether animals behave optimally is an open question of great importance, both theoretically and in practice. Attempts to answer this question focus on two aspects of the optimization problem, the quantity to be optimized and the optimization process itself. In this paper, we assume the abstract concept of cost as the quantity to be minimized and propose a reinforcement learning algorithm, called Value-Gradient Learning (VGL), as a computational model of behavior optimality. We prove that, unlike standard models of Reinforcement Learning, Temporal Difference in particular, VGL is guaranteed to converge to optimality under certain conditions. The core of the proof is the mathematical equivalence of VGL and Pontryagin's Minimum Principle, a well-known optimization technique in systems and control theory. Given the similarity between VGL's formulation and regulatory models of behavior, we argue that our algorithm may provide psychologists with a tool to formulate such models in optimization terms.

show abstract

Section: Proof Of Optimalitymentioning

confidence: 64%

Back to optimality: a formal framework to express the dynamics of learning optimal behavior

2015

Self Cite

View full text Add to dashboard Cite

show abstract

“…But again, due to the large state space, value function approximation is a necessity, violating the assumptions for guaranteed convergence and thus leaving room for asymptotic performance gains as well. The authors in [25,26] have shown that DHP algorithms can eventually find an optimal solution without the explicit need for stochastic exploration, but the value learning algorithms (i.e. TD, TDp0q) could not.…”

Section: Related Workmentioning

confidence: 99%

“…r Ω is a ball of radius r R . By taking the time derivative of the first term with respect to the state trajectories with uptq (see, (37)) and the second term with respect to the perturbed critic estimation error dynamics (23), using (25), substituting the update for the actor (31) and grouping terms together, then (40) becomes,…”

Section: Proof Of Theoremmentioning

confidence: 99%

Asymptotically Stable Adaptive–Optimal Control Algorithm With Saturating Actuators and Relaxed Persistence of Excitation

Vamvoudakis

Miranda

Hespanha

2016

IEEE Trans. Neural Netw. Learning Syst.

158

View full text Add to dashboard Cite

This paper proposes a control algorithm based on adaptive dynamic programming to solve the infinite-horizon optimal control problem for known deterministic nonlinear systems with saturating actuators and nonquadratic cost functionals. The algorithm is based on an actor/critic framework, where a critic neural network (NN) is used to learn the optimal cost, and an actor NN is used to learn the optimal control policy. The adaptive control nature of the algorithm requires a persistence of excitation condition to be a priori validated, but this can be relaxed using previously stored data concurrently with current data in the update of the critic NN. A robustifying control term is added to the controller to eliminate the effect of residual errors, leading to the asymptotically stability of the closed-loop system. Simulation results show the effectiveness of the proposed approach for a controlled Van der Pol oscillator and also for a power system plant.

show abstract

“…Because HDP is an algorithm which requires stochastic exploration to optimise the ADP/RL problem effectively [32], in the HDP experiment we had to modify (3) to choose exploratory actions. Hence for the HDP experiment we used…”

Section: A Vertical-lander Problemmentioning

confidence: 99%

Clipping in Neurocontrol by Adaptive Dynamic Programming

Fairbank

Prokhorov²,

Alonso

2014

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

This is the accepted version of the paper.This version of the publication may differ from the final published version. Abstract-In adaptive dynamic programming, neurocontrol and reinforcement learning, the objective is for an agent to learn to choose actions so as to minimise a total cost function. In this paper we show that when discretized time is used to model the motion of the agent, it can be very important to do "clipping" on the motion of the agent in the final time step of the trajectory. By clipping we mean that the final time step of the trajectory is to be truncated such that the agent stops exactly at the first terminal state reached, and no distance further. We demonstrate that when clipping is omitted, learning performance can fail to reach the optimum; and when clipping is done properly, learning performance can improve significantly. The clipping problem we describe affects algorithms which use explicit derivatives of the model functions of the environment to calculate a learning gradient. These include Backpropagation Through Time for Control, and methods based on Dual Heuristic Programming. However the clipping problem does not significantly affect methods based on Heuristic Dynamic Programming, Temporal Differences Learning or Policy-Gradient Learning algorithms. Permanent repository link

show abstract

A comparison of learning speed and ability to cope without exploration between DHP and TD(0)

Cited by 8 publications

References 18 publications

Back to optimality: a formal framework to express the dynamics of learning optimal behavior

Back to optimality: a formal framework to express the dynamics of learning optimal behavior

Asymptotically Stable Adaptive–Optimal Control Algorithm With Saturating Actuators and Relaxed Persistence of Excitation

Clipping in Neurocontrol by Adaptive Dynamic Programming

Contact Info

Product

Resources

About