A centralised training algorithm with D3QN for scalable regular unmanned ground vehicle formation maintenance

Abstract

The unmanned ground vehicle (UGV) has been widely used to accomplish various missions in civilian or military environments. Formation of the UGVs group is an important technique to support the broad applications of multi-functional UGVs. This study proposes a scalable regular UGV formation maintenance (SRUFM) algorithm based on deep reinforcement learning (DRL), which aims to use a unified DRL framework to improve the lateral control and longitudinal control performance of UGV in different situations of the formation. Based on the tailored design of state, action space, related inter-vehicle information, and the dueling double deep Q-network (D3QN), SRUFM outperforms nature deep Q-network and double deep Q-network in the exploration efficiency and the convergence speed in same CARLA training environments with fixed size formation. Furthermore, when the formation's scale is extended with similar initialisation conditions, the SRUFM can still get a nearly 90% success rate to execute all experiment formation maintenance missions after 4000 episodes of training. Each UGV in the formation can keep distance within the upper and lower error threshold of 0.15 m. The simulation experiments show that the proposed centralised training frame with D3QN is suitable to solve scalable regular UGV formation maintenance missions.

Multi-objective optimization of cloud task scheduling based on improved deep Q-learning algorithm

“…Traditional Q learning [10] belongs to the value-based algorithm of reinforcement learning algorithm,Q that is,Q (s,a),is the state state at a certain moment,take action a can get the expectation of gain,the environment will feedback the corresponding reward reward according to the agent's action,so the main idea of the algorithm is to build the state and action into a Qtable table to store Q values,and then select the action that can obtain the maximum benefit according to Q values.As the complexity of the environment increases,Qtable is difficult to adapt to tasks with huge state space.2013 DeepMind team [11]proposed the DQN algorithm,which combines deep learning and reinforcement learning for the first time.In 2016,Tom Schaul [12] proposed a preferred experience playback method,which uses temporal difference error to measure the learning value of each experience; secondly,the absolute value of temporal difference error is used to rank the experiences in the experience pool,and the experiences with high deviation are played back more frequently,while the importance sampling weights to correct the high bias problem,which speeds up the training and reduces the convergence difficulty.In 2016 Van Hasselt [13] proposed DoubleDQN to provide a solution to the problem of DQN overestimation,by implementing the selection of actions and the evaluation of actions with different value functions.Wang [14] further proposed DuelingDQN,which The dominance function "normalizes" the Q-Value to the Value baseline,which helps to improve the learning efficiency and make the learning more stable; at the same time,experience shows that the dominance function also helps to reduce the variance,which is an important factor of overfitting.However,D3QN [15] combined with prioritized empirical replay still has shortcomings,and the ability to explore the optimal path is still weak.…”

Driver-like decision-making method for vehicle longitudinal autonomous driving based on deep reinforcement learning

“…Therefore, many researchers used deep reinforcement learning methods to study vehicle decisionmaking problems. [12][13][14] The deep reinforcement learning methods replaces the value function of the continuous state space with a neural network, and outputs discrete (such as DQN algorithm) or continuous actions (such as DDPG algorithm), which reduces computer memory requirement, improves decision-making accuracy and training efficiency, and improves traffic efficiency and safety, reduces fuel consumption reduction. Studies have shown that deep reinforcement learning methods are more excellent than traditional methods, [15][16][17] and can get good effects in complex environments.…”