Model Imitation for Model-Based Reinforcement Learning

Wu, Yueh-Hua; Fan, Ting-Han; Ramadge, Peter J.; Sheng, Hao

doi:10.48550/arxiv.1909.11821

Cited by 2 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mishra et al [14] proposed the temporal segments model to make the prediction of the agent more accurate and stable. Wu et al [29] tried to use the WGAN learning transition model by matching the multi-step rollouts distributions and the real ones to reduce the estimation error of the model. Besides, on the basis of the Model-based Value Expansion (MVE) proposed by Feinberg et al [6], Xiao et al [30] proposed Adaptive Model-based Value Expansion (AdaMVE).…”

Section: Dyna-style Algorithmmentioning

confidence: 99%

Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning

Wang¹,

Zhang²,

Huang³

et al. 2020

Preprint

View full text Add to dashboard Cite

Model-based reinforcement learning is a framework in which an agent learns an environment model, makes planning and decision-making in this model, and finally interacts with the real environment. Model-based reinforcement learning has high sample efficiency compared with model-free reinforcement learning, and shows great potential in the real-world application. However, model-based reinforcement learning has been plagued by dynamics bottleneck. Dynamics bottleneck is the phenomenon that when the timestep to interact with the environment increases, the reward of the agent falls into the local optimum instead of increasing. In this paper, we analyze and explain how the coupling relationship between model and policy causes the dynamics bottleneck and shows improving the exploration ability of the agent can alleviate this issue. We then propose a new planning algorithm called Maximum Entropy Cross-Entropy Method (MECEM). MECEM can improve the agent's exploration ability by maximizing the distribution of action entropy in the planning process. We conduct experiments on fourteen wellrecognized benchmark environments such as HalfCheetah, Ant and Swimmer. The results verify that our approach obtains the state-of-the-art performance on eleven benchmark environments and can effectively alleviate dynamics bottleneck on HalfCheetah, Ant and Walker2D.

show abstract

Section: Dyna-style Algorithmmentioning

confidence: 99%

Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning

Wang¹,

Zhang²,

Huang³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…As a result, the robot may make incorrect decisions and take sub-optimal actions when solving largescale decision-making control problems, leading to a degradation in asymptotic performance [12]. At present, various methods have been proposed to alleviate compounding model error (Wu et al [13]; Janner et al [14]; Lai et al [15]). For example, Janner et al [14] proposed MBPO (Model-Based Policy Optimization) to use truncated short-rollouts branched from real states.…”

Section: Introductionmentioning

confidence: 99%

Bidirectional Model-Based Policy Optimization Based on Adaptive Gaussian Noise and Improved Confidence Weights

Liu,

Jin

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Model-Based Reinforcement Learning (MBRL) has been gradually applied in the field of Robot Learning due to its excellent sample efficiency and asymptotic performance. However, for high-dimensional learning tasks in complex scenes, the exploration and stable training capabilities of the robot still need enhancement. In light of policy planning and policy optimization, we propose a bidirectional model-based policy optimization algorithm based on adaptive gaussian noise and improved confidence weights (BMPO-NW). The algorithm parameterizes bidirectional policy networks into noise networks by adding different adaptive Gaussian noises to the connection weights and biases. This can improve the randomness of policy search and induce efficient exploration for the robot. Simultaneously, the confidence weight of improved activation function is introduced into the Q-function update formula of SAC, which can reduce the error propagation problem of target Q-network, and enhance the robot's training stability. Finally, we implement the improved algorithm based on the framework of bidirectional model-based policy optimization algorithm (BMPO) to ensure asymptotic performance and sample efficiency. Experimental results in MuJoCo benchmark environments demonstrate that the learning speed of BMPO-NW is about 20% higher than baseline methods, the average reward is about 15% higher than other MBRL methods, and 50%-70% higher than MFRL methods, while the training process is more stable. Ablation experiments and different variant design experiments further verify the feasibility and robustness. The research results provide theoretical support for the conclusion of this paper and hold significant practical value for MBRL to help the robot realize applications in complex scenarios.

show abstract

Model Imitation for Model-Based Reinforcement Learning

Cited by 2 publications

References 9 publications

Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning

Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning

Bidirectional Model-Based Policy Optimization Based on Adaptive Gaussian Noise and Improved Confidence Weights

Contact Info

Product

Resources

About