Dyna-MLAC: Trading Computational and Sample Complexities in Actor-Critic Reinforcement Learning

Costa, Bruno Sielly Jales; Caarls, Wouter; Menasché, Daniel Sadoc

doi:10.1109/bracis.2015.62

Cited by 2 publications

(1 citation statement)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Enormous samples are still required when only using such a process model to update the policy gradient. Afterward, Costa et al [ 26 ] derived an AC algorithm by introducing Dyna structure called Dyna-MLAC which approximated the value function, the policy, and the model by LLR as MLAC did. The difference is that Dyna-MLAC applies the model not only in updating the policy gradient but also in planning [ 27 ].…”

Section: Introduction and Related Workmentioning

confidence: 99%

Efficient Actor-Critic Algorithm with Hierarchical Model Learning and Planning

Zhong

Liu

2016

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

To improve the convergence rate and the sample efficiency, two efficient learning methods AC-HMLP and RAC-HMLP (AC-HMLP with ℓ 2-regularization) are proposed by combining actor-critic algorithm with hierarchical model learning and planning. The hierarchical models consisting of the local and the global models, which are learned at the same time during learning of the value function and the policy, are approximated by local linear regression (LLR) and linear function approximation (LFA), respectively. Both the local model and the global model are applied to generate samples for planning; the former is used only if the state-prediction error does not surpass the threshold at each time step, while the latter is utilized at the end of each episode. The purpose of taking both models is to improve the sample efficiency and accelerate the convergence rate of the whole algorithm through fully utilizing the local and global information. Experimentally, AC-HMLP and RAC-HMLP are compared with three representative algorithms on two Reinforcement Learning (RL) benchmark problems. The results demonstrate that they perform best in terms of convergence rate and sample efficiency.

show abstract

Section: Introduction and Related Workmentioning

confidence: 99%