Improve exploration in deep reinforcement learning for UAV path planning using state and action entropy

Abstract

Despite being a widely adopted development framework for unmanned aerial vehicle (UAV), deep reinforcement learning (DRL) is often considered sample inefficient. Particularly, UAV struggles to fully explore the state and action space in environments with sparse rewards. While some exploration algorithms have been proposed to overcome the challenge of sparse rewards, they are not specifically tailored for UAV platform. Consequently, applying those algorithms to UAV path planning may lead to problems such as unstable training processes and neglect of action space comprehension, possibly causing negative impacts on the path planning results. To address the problem of sparse rewards in UAV path planning, we propose an information-theoretic exploration algorithm, Entropy Explorer (EE), specifically for UAV platform. The proposed EE generates intrinsic rewards based on state entropy and action entropy to compensate for the scarcity of extrinsic rewards. To further improve sampling efficiency, a framework integrating EE and Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithms is proposed. Finally, the TD3-EE algorithm is tested in Airsim and compared against benchmarking algorithms. The simulation outcomes manifest that TD3-EE effectively stimulates the UAV to comprehensively explore both state and action spaces, thereby attaining superior performance compared to the benchmark algorithms in the realm of path planning.

Self-supervised monocular visual odometry based on cross-correlation

“…where, x and y represent both the corresponding coordinates of the elements in the cross-correlation matrix φ and the coordinates of the cross-correlation elements in V t and V t+i , respectively, and M denotes the cross-correlation module. We use the Euclidean distance of the corresponding patches in the high-dimensional feature space as a across-correlation metric, as shown in equation (3), where the smaller the distance is, the higher the degree of cross-correlation and the smaller the difference of the corresponding patches…”

“…In unknown and complex environments, visual odometry [1] plays a crucial role as a key technology for motion estimation and self-localization solely based on the onboard sensors of vehicles [2], drones [3] and robots [4]. Substantial research efforts [5,6] have been dedicated to devising a visual odometry estimation system that integrates accuracy, robustness and from consecutive images, followed by estimating relative pose between two adjacent frames based on geometric relationships.…”

Research on mobile robot path planning in complex environment based on DRQN algorithm

“…The second layer of the RBF network is the hidden layer with s nodes and the basis function is j k (Y) (k = 1, 2, L, T, G). The expression of the t-th node in this layer based on the G-dimensional Gaussian function is shown in equation (19).…”

“…Deep reinforcement learning combines the perceptual ability of deep learning and the decision-making ability of reinforcement learning, which effectively solves the problems of dimensional catastrophe and algorithm training in Reinforcement Learning [19], and provides a new idea for solving path-planning problems in complex environments. Deep Q learning [20], Deep Deterministic Policy Gradient (DDPG) [21], Twin Delayed Deep Deterministic Policy Gradient [22], and other methods have been used in the fields of mobile robot task allocation and control optimization.…”