Control and Simulation of a 6-DOF Biped Robot based on Twin Delayed Deep Deterministic Policy Gradient Algorithm

Abstract

Objectives:To study an algorithm to control a bipedal robot to walk so that it has a gait close to that of a human. It is known that the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is a highly efficient algorithm with a few changes compared to the popular algorithm -the commonly used Deep Deterministic Policy Gradient (DDPG) in the continuous action space problem in Reinforcement Learning. Methods: Different from the usual sparse reward function model used, in this study, a reward model combined with a sparse reward function and dense reward function will be proposed. The application of the TD3 algorithm together with the proposed reward function model to control a bipedal robot model with 6 degrees of freedom will be presented. The training process is simulated in Gazebo/Robot Operating System (ROS) environment. Finding: The results show that, when choosing a reward model combined with a sparse reward function and a dense reward function suitable for the robot model, will help it learn faster and achieve better results. The biped robot can walk straight with an almost human-like gait. In the paper, the results from the TD3 algorithm combined with the proposed reward model are also compared with the results from other algorithms. Novelty: Applying the TD3 algorithm combined with the proposed reward model for the 6-DOF biped robot and simulating the robot's gait in Gazebo/ROS environment, ROS is a middleware that can be used to control a robot in a real environment in the future.

Review of Reinforcement Learning for Robotic Grasping: Analysis and Recommendations

“…To address this issue, M et al combined TD3 with Hindsight Experience Replay (HER) [165]. Khoi et al utilized the TD3 algorithm along with a novel reward model to simulate the gait of a 6-DOF biped robot in a Gazebo/ROS environment [166]. Yang and Xu aimed to design a robot that can aid in warehouse object grasping using various DRL algorithms, including TD3 [167].…”

A Trajectory Control for Bipedal Walking Robot Using Stochastic-Based Continuous Deep Reinforcement Learning

“…P.B. Khoi et al presented Twin Delayed DDPG (TD3) 1) to control the bipedal walking robot. TD3 can handle the overestimation bias on DDPG, but TD3 is still using the deterministic policy.…”

GCTD3: Modeling of Bipedal Locomotion by Combination of TD3 Algorithms and Graph Convolutional Network

“…In this research, an extension of the TD3 [20] algorithm was proposed to include more information about the connection between the joints of the robot in the training process. In fact, there are many articles [20][21][22][23][24][25] using reinforcement learning algorithms such as TD3, DDPG and SAC to find the desired angle values of the joints of the robot. However, their algorithms only used the information about the velocity and angular value of the joints for training, they did not take advantage of the graph topology and the binding relationship of the humanoid robot, as in our method.…”

“…In each state, the height of the body robot has different values. In the paper [25], only an average value of the body height during motion is used as a basis height for the robot to learn, two average values of the body height corresponding to two grounding states of the legs in motion are used in this paper. At the single phase of walking (Figure 7a,b), the average height of the robot's body reaches a higher value than that at the double phase of walking (Figure 7c,d).…”