A Framework for DRL Navigation With State Transition Checking and Velocity Increment Scheduling

Abstract

To train a mobile robot to navigate using end-to-end approach which maps sensors data into actions, we can use deep reinforcement learning (DRL) method by providing training environments with proper reward functions. Although some studies have shown the success of DRL in navigation task for mobile robots, the method needs appropriate hyperparameter settings such as the environment's timestep size and the robot's velocity range to produce a good navigation policy. The previous existing DRL framework has proposed the use of odometry sensor to generate dynamic timestep size in the environment to solve the mismatch problem between the timestep size and the robot's velocity. However, the framework lacks a procedure for checking terminal conditions which may occur during action executions resulting inconsistency in the environment and poor navigation policies. In the case of navigation task, terminal conditions may happen when the robot achieves the navigation goal position or collides with obstacles while performing an action in one timestep. To cope with this problem, we propose a state transition checking method in the DRL environment which is specific for navigation task that leverages odometry and laser sensor to ensure that the environment follows Markov Decision Process with dynamic timestep size. We also introduce a velocity increment scheduling to stabilize the mobile robot during training. Our experiment results show that state transition checking along with the velocity increment scheduling are able to make the robot navigate faster with higher success rate compared to other existing DRL frameworks.

Integrating Multiple Policies for Person-Following Robot Training Using Deep Reinforcement Learning

“…As also shown in Table 1, we use different reward functions inside the navigation and the attending environment since all environments have dissimilar objectives. We employ a reward function which is based on the artificial potential field (APF) inside the navigation environment so that the robot is capable of avoiding obstacles and able to reach the target person [15], [42], [43]. On the other hand, we apply U-Shaped reward function [22] inside the attending environment so that the robot is able to attend the target person at his left or at his right side as depicted in Fig.…”

“…Since we have already obtained the optimal navigation policy π nav from our prior study [15], we do not need to perform the training for the navigation task anymore. Instead, we reuse and integrate the policy to the left attending policy and the right attending policy which are obtained in this study.…”

“…6 (c)) for the person-following task validation. In addition, all environments in this study are implemented in Python which follow the standardized Ope-nAI Gym framework [47] and the state transition checking method [15].…”

“…The goal of our study is to train a mobile robot capable of performing the person-following task using DRL method. Since we have already obtained the optimal navigation policy from our previous work [15] and we want to reuse the policy in this study, we then extend the prior work of Frans et al [14] so that it is capable of reusing optimal policies which correspond to all sub tasks in a complex environment and have been trained beforehand. The main contributions of the paper are summarized as follows:…”

Resource Allocation for Multi-Tenant Network Slicing: A Multi-Leader Multi-Follower Stackelberg Game Approach

“…An environment that follows an MDP is needed to train DRL agents [43], which is formulated by conceptualizing the dynamic relationship between the RSUs and UAVs as a Stackelberg game. This strategic interaction is then formally cast as a multi-agent MDP.…”