Online Trajectory Planning Method for Midcourse Guidance Phase Based on Deep Reinforcement Learning

Abstract

Concerned with the problem of interceptor midcourse guidance trajectory online planning satisfying multiple constraints, an online midcourse guidance trajectory planning method based on deep reinforcement learning (DRL) is proposed. The Markov decision process (MDP) corresponding to the background of a trajectory planning problem is designed, and the key reward function is composed of the final reward and the negative step feedback reward, which lays the foundation for the interceptor training trajectory planning method in the interactive data of a simulation environment; at the same time, concerned with the problems of unstable learning and training efficiency, a trajectory planning training strategy combined with course learning (CL) and deep deterministic policy gradient (DDPG) is proposed to realize the progressive progression of trajectory planning learning and training from satisfying simple objectives to complex objectives, and improve the convergence of the algorithm. The simulation results show that our method can not only generate the optimal trajectory with good results, but its trajectory generation speed is also more than 10 times faster than the hp pseudo spectral convex method (PSC), and can also resist the error influence mainly caused by random wind interference, which has certain application value and good research prospects.

DDPG-Based Convex Programming Algorithm for the Midcourse Guidance Trajectory of Interceptor

“…As a relatively mature algorithm in deep reinforcement learning, the DDPG algorithm has significant advantages over other deep reinforcement learning algorithms (such as deep Q network (DQN), deterministic policy gradient (DPG), etc.) in handling continuous action spaces, efficient gradient optimization, utilizing experience replay buffers, and improving stability [37]. This makes the DDPG algorithm achieve higher performance and efficiency in solving complex continuous control tasks.…”

“…During the midcourse guidance phase, the interceptor flies at high altitude and high speed for a long time, which is subject to constraints on heat flux density Q, dynamic pressure p, overload n, angle, and control variables. Therefore, the following constraints should also be met [37].…”

“…The trained strategy function is the final initial solution trajectory fast generation method. The specific model of the DDPG algorithm used in this paper and the interceptor's guidance motion model used for the interaction between the agent and the environment can be found in Ref [37], and they will not be detailed here due to space limitations. Figure 2 shows the training framework for the trajectory planning task based on the DDPG algorithm.…”

Trajectory Optimization for the Nonholonomic Space Rover in Cluttered Environments Using Safe Convex Corridors

“…Over the past few decades, substantial progress has been made in motion planning methods for nonholonomic robots, including space rovers. These methods encompass the polynomial interpolation method [5,6], adaptive state lattices [7], homotopy-based methods [8], probabilistic search methods such as rapidly exploring random tree [9], informed RRT* [10], fast marching trees [11,12], reinforcement learning method [13][14][15], numerical optimization methods [16][17][18][19], and others.…”