UAV Motion Strategies in Uncertain Dynamic Environments: A Path Planning Method Based on Q-Learning Strategy

Abstract

A solution framework for UAV motion strategies in uncertain dynamic environments is constructed in this paper. Considering that the motion states of UAV might be influenced by some dynamic uncertainties, such as control strategies, flight environments, and any other bursting-out threats, we model the uncertain factors that might cause such influences to the path planning of the UAV, unified as an unobservable part of the system and take the acceleration together with the bank angle of the UAV as a control variable. Meanwhile, the cost function is chosen based on the tracking error, then the control instructions and flight path for UAV can be achieved. Then, the cost function can be optimized through Q-learning, and the best UAV action sequence for conflict avoidance under the moving threat environment can be obtained. According to Bellman’s optimization principle, the optimal action strategies can be obtained from the current confidence level. The method in this paper is more in line with the actual UAV path planning, since the generation of the path planning strategy at each moment takes into account the influence of the UAV control strategy on its motion at the next moment. The simulation results show that all the planning paths that are created according to the solution framework proposed in this paper have a very high tracking accuracy, and this method has a much shorter processing time as well as a shorter path it can create.

High-Level Path Planning for an Autonomous Sailboat Robot Using Q-Learning

“…Jaradat and Jiang used only eight actions to represent the robot movement while Khriji used 14 actions. Cui et al [31] and Zhang et al [32] used modified Q-Learning approaches to solve the problem of path-planning and obstacle avoidance for UAVs. The idea is that since a UAV acts in a dynamic and uncertain environment subjected to wind forces, state estimation and actions are usually prone to errors.…”

2D Optimal Trajectory Planning Problem in Threat Environment for UUV with Non-Uniform Radiation Pattern

“…Additionally, an approach based on ant-colony behavior is often used [ 11 , 17 ]. In recent years, due to the progress of neural networks, the deep learning and specifically deep reinforcement learning approaches [ 18 , 19 ] are becoming more and more popular.…”

3D real-time path planning of UAVs in dynamic environments in the presence of uncertainty

“…27 In addition UAV state uncertainty including both perception and dynamics shall also be considered. 16,21 Different studies tried to incorporate uncertainty modelling within their path planning systems. Although as will be discussed in the next sub-sections different approaches have been made to eliminate or heavily attenuate uncertainty effects on path planning systems Kim et.…”

“…considers UAV dynamic path planning as extremely challenging due to different uncertainties, magnitude fuzzy and interaction of these factors and constraints present within UAV utilisation environments. 16 The planner has to deal with errors and imperfections of sensing systems to figure out threats and positional, kinematics and dynamics information for state estimation in real-time. Moreover, UAV mission objectives and control modes of operation of UAVs further add to the complexity of the problem.…”