A Hierarchical Framework for Quadruped Robots Gait Planning Based on DDPG

Abstract

In recent years, significant progress has been made in employing reinforcement learning for controlling legged robots. However, a major challenge arises with quadruped robots due to their continuous states and vast action space, making optimal control using simple reinforcement learning controllers particularly challenging. This paper introduces a hierarchical reinforcement learning framework based on the Deep Deterministic Policy Gradient (DDPG) algorithm to achieve optimal motion control for quadruped robots. The framework consists of a high-level planner responsible for generating ideal motion parameters, a low-level controller using model predictive control (MPC), and a trajectory generator. The agents within the high-level planner are trained to provide the ideal motion parameters for the low-level controller. The low-level controller uses MPC and PD controllers to generate the foot-end force and calculates the joint motor torque through inverse kinematics. The simulation results show that the motion performance of the trained hierarchical framework is superior to that obtained using only the DDPG method.

Special Issue: Design and Control of a Bio-Inspired Robot

“…Experimental results using the BHR7P bipedal robot validate the effectiveness of these proposed methods. The paper by Yanbiao [4] presents a hierarchical reinforcement learning framework based on the Deep Deterministic Policy Gradient (DDPG) algorithm. This framework involves a high-level planner that generates ideal motion parameters, a lowlevel controller using model predictive control (MPC), and a trajectory generator.…”

An improved DDPG algorithm based on evolution-guided transfer in reinforcement learning

“…It can be adept at dealing with the high-dimensional action spaces commonly encountered in real-world applications like robotics, autonomous vehicles, and complex control systems. The soft target updates employed by DDPG also contribute to more robust and reliable learning, including controlling a car's throttle [25] or a robot's joint angles [26]. Furthermore, the improved version of DDPG has been proposed to enhance learning efficiency.…”

Biped Robots Control in Gusty Environments with Adaptive Exploration Based DDPG

“…To enhance the adaptability of these learning strategies while reducing their computational demands, current research trends towards integrating reinforcement learning with conventional control methods [ 17 ]. For example, Li and colleagues developed a hierarchical framework for quadruped robots’ gait planning that combines the DDPG algorithm with Model Predictive Control (MPC), achieving optimal action control [ 18 ]. Moreover, since the introduction of the soft actor-critic (SAC) algorithm, reinforcement learning-based gait control methods have made significant strides in the field of robotics [ 19 ].…”