Deep Reinforcement Learning With Optimized Reward Functions for Robotic Trajectory Planning

This paper proposes an improved Q-learning method to obtain near-optimal schedules for grid and battery power in a grid-connected electric vehicle charging station for a 24-hour horizon. The charging station is supplied by a solar PV generator with a backup from the utility grid. The grid tariff model is dynamic in line with the smart grid paradigm. First, the mathematical formulation of the problem is developed highlighting each of the cost components considered including battery degradation cost and the real-time tariff for grid power purchase cost. The problem is then formulated as a Markov Decision Process (MDP), i.e., defining each of the parts of a reinforcement learning environment for the charging station's operation. The MDP is solved using the improved Q-learning algorithm proposed in this paper and the results are compared with the conventional Q-learning method. Specifically, the paper proposes to modify the action-space of a Q-learning algorithm so that each state has just the list of actions that meet a power balance constraint. The Q-table updates are done asynchronously, i.e., the agent does not sweep through the entire state-space in each episode. Simulation results show that the improved Q-learning algorithm returns a 14% lower global cost and achieves higher total rewards than the conventional Q-learning method. Furthermore, it is shown that the improved Q-learning method is more stable in terms of the sensitivity to the learning rate than the conventional Q-learning.

show abstract

“…2) Negative reward function, i.e., reward expressed as negative of the cost as used in [28] and is given by. − ( ( ) + ( )) ( 11)…”

Section: Rewardmentioning

confidence: 99%

Improved Q-learning for Energy Management in a Grid-tied PV Microgrid

Arwa

Folly

2021

SAIEE Afr. Res. J.

View full text Add to dashboard Cite

show abstract

“…In such cases, which entail most realistic applications, the value function needs to be approximated. In a large group of works, e.g., [10]- [14], artificial neural networks were employed to approximate the value function over the entire state-space. Despite few successful trials, most early attempts of such were not very successful due to the overfitting problem.…”

Section: Introductionmentioning

confidence: 99%

MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learning

et al. 2020

View full text Add to dashboard Cite

Background: There has been an increasing surge of interest on development of advanced Reinforcement Learning (RL) systems as intelligent approaches to learn optimal control policies directly from smart agents' interactions with the environment. Objectives: In a model-free RL method with continuous state-space, typically, the value function of the states needs to be approximated. In this regard, Deep Neural Networks (DNNs) provide an attractive modeling mechanism to approximate the value function using sample transitions. DNN-based solutions, however, suffer from high sensitivity to parameter selection, are prone to overfitting, and are not very sample efficient. A Kalman-based methodology, on the other hand, could be used as an efficient alternative. Such an approach, however, commonly requires a-priori information about the system (such as noise statistics) to perform efficiently. The main objective of this paper is to address this issue. Methods: As a remedy to the aforementioned problems, this paper proposes an innovative Multiple Model Kalman Temporal Difference (MM-KTD) framework, which adapts the parameters of the filter using the observed states and rewards. Moreover, an active learning method is proposed to enhance the sampling efficiency of the system. More specifically, the estimated uncertainty of the value functions are exploited to form the behaviour policy leading to more visits to less certain values, therefore, improving the overall learning sample efficiency. As a result, the proposed MM-KTD framework can learn the optimal policy with significantly reduced number of samples as compared to its DNN-based counterparts. Results: To evaluate performance of the proposed MM-KTD framework, we have performed a comprehensive set of experiments based on three RL benchmarks, namely, Inverted Pendulum; Mountain Car, and; Lunar Lander. Experimental results show superiority of the proposed MM-KTD framework in comparison to its state-of-the-art counterparts.

show abstract

“…Another key part is the design of reward function since the optimization of strategy is based on the cumulative reward of each state. How to design the reward function will affect the efficiency and effectiveness of skill learning in the framework of robot reinforcement learning [10]. It is still a difficult task to design a suitable reward function, since different tasks are often difficult to have a unified reward function.…”

Section: Introductionmentioning

confidence: 99%

A Reinforcement Learning-Based Framework for Robot Manipulation Skill Acquisition

Liu

Wang

et al. 2020

IEEE Access

View full text Add to dashboard Cite

This paper studies robot manipulation skill acquisition based on a proposed reinforcement learning framework. Robot can learn policy autonomously by interacting with environment with a better learning efficiency. Aiming at the manipulator operation task, a reward function design method based on objects configuration matching (OCM) is proposed. It is simple and suitable for most Pick and Place skills learning. Integrating robot and object state, high-level action set and the designed reward function, the Markov model of robot manipulator is built. An improved Proximal Policy Optimize algorithm with manipulation set as the output of Actor (MAPPO) is proposed as the main structure to construct the robot reinforcement learning framework. The framework combines with the Markov model to learn and optimize the skill policy. A same simulation environment as the real robot is set up, and three robot manipulation tasks are designed to verify the effectiveness and feasibility of the reinforcement learning framework for skill acquisition.

show abstract

Deep Reinforcement Learning With Optimized Reward Functions for Robotic Trajectory Planning

Cited by 61 publications

References 25 publications

Improved Q-learning for Energy Management in a Grid-tied PV Microgrid

Improved Q-learning for Energy Management in a Grid-tied PV Microgrid

MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learning

A Reinforcement Learning-Based Framework for Robot Manipulation Skill Acquisition

Contact Info

Product

Resources

About