Non-Communication Decentralized Multi-Robot Collision Avoidance in Grid Map Workspace with Double Deep Q-Network

Abstract

This paper presents a novel decentralized multi-robot collision avoidance method with deep reinforcement learning, which is not only suitable for the large-scale grid map workspace multi-robot system, but also directly processes Lidar signals instead of communicating between the robots. According to the particularity of the workspace, we handcrafted a reward function, which considers both the collision avoidance among the robots and as little as possible change of direction of the robots during driving. Using Double Deep Q-Network (DDQN), the policy was trained in the simulation grid map workspace. By designing experiments, we demonstrated that the learned policy can guide the robot well to effectively travel from the initial position to the goal position in the grid map workspace and to avoid collisions with others while driving.

Non-communicating Decentralized Multi-robot Collision Avoidance in Grid Graph Workspace based on Dueling Double Deep Q-Network

“…Since the application scenarios of this paper are the same as those proposed by [10], and the algorithm selection is consistent; therefore, the network structure selection is compared with them. Based on Chen et al's network structure, we introduced LSTM layer to obtain temporal information of data in order to enable the robot to make decisions based on historical information, and we added CBAM hybrid attention mechanism [12] in order to enable the robot to pay more attention to the information of nearby robots.…”

“…For the scenario in this paper, the robot can only get the reward when it reaches the end point or collision. For this sparse reward scenario, adding the appropriate process reward can speed up the training process.Therefore, based on the reward of [10], we design a new reward function for this motion model, and we still judge the effectiveness of the reward function in terms of two metrics, 𝑅 𝑎𝑐𝑐 and 𝑅 𝑚𝑜𝑣𝑒 . We demonstrate the effectiveness of the reward function through two sets of experiments, one using the new reward function designed in this paper to derive policy2 (i.e., policy2 above), and the other not using the reward function in this paper to derive policy3, using the same network structure as [10], and keeping the other parameters the same for 10,000 rounds.…”

“…Chen et al [10] has proposed a novel decentralized multi-robot collision avoidance method with deep reinforcement learning, which utilizes Lidar signals directly and does not require communication between robots. However, when there are static obstacles in the scene, it is difficult to extract more useful information from the laser signal, resulting in low obstacle avoidance performance and task execution efficiency.…”

FishGym: A High-Performance Physics-based Simulation Framework for Underwater Robot Learning

“…For example, PPO was used in multi-robot collision avoidance task [66], bipedal robot locomotion [67] etc. DQN also proves to work well on a certain type of tasks in real robots [68]- [70]. Recently, SAC [71] emerges to combine the strengths of the above two main approaches and has proved its capability in real robot problems like Dexterous manipulation [72], mobile robot navigation [73], robot arm control [74], multi-legged robot [75], etc.…”

Prediction of Soil Heavy Metal Content Based on Deep Reinforcement Learning

“…In the adaptive inverse distance weighted method, hyperparameters of each known point in the model are learned, and the nearest adjacent statistics of each point are calculated. Furthermore, the multidimensional spatial discrete points are formed, and spatial modeling is done by Kriging interpolation method [19,20]. Finally, the corresponding coordinates of interpolation points to be predicted are input into the spatial model, so as to obtain the corresponding hyperparameters of interpolation points.…”