Deep Reinforcement Learning Based High-level Driving Behavior Decision-making Model in Heterogeneous Traffic

Abstract

High-level driving behavior decision-making is an open-challenging problem for connected vehicle technology, especially in heterogeneous traffic scenarios. In this paper, a deep reinforcement learning based high-level driving behavior decision-making approach is proposed for connected vehicle in heterogeneous traffic situations. The model is composed of three main parts: a data preprocessor that maps hybrid data into a data format called hyper-grid matrix, a two-stream deep neural network that extracts the hidden features, and a deep reinforcement learning network that learns the optimal policy. Moreover, a simulation environment, which includes different heterogeneous traffic scenarios, is built to train and test the proposed method. The results demonstrate that the model has the capability to learn the optimal high-level driving policy such as driving fast through heterogeneous traffic without unnecessary lane changes. Furthermore, two separate models are used to compare with the proposed model, and the performances are analyzed in detail.

Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles

“…In the area of motion planning, the end episode rewards are calculated from the fulfillment or failure of the driving task. The overall performance factors are generally: time of finishing the task, keeping the desired speed or achieving as high average speed as possible, yaw or distance from lane middle or the desired trajectory, overtaking more vehicles, achieve as few lane changes as possible [57], keeping right [58], [59] etc. Rewarding systems also can represent passenger comfort, where the smoothness of the vehicle dynamics is enforced.…”

“…surrounding vehicles in a grid needs not only occupancy, but other information hence the spatial grid's cell need to hold additional information. In [57] the authors used equidistant grid, where the egovehicle is placed in the center, and the cells occupied by other vehicles represented the longitudinal velocity of the corresponding car (See Fig. 7).…”

“…The realism of the models could still differentiate endto-end solutions. For example, in [57], instead of using the nonholonomic Ackermann steering geometry, the authors use a holonomic robot model for the action space, which highly reduces the complexity of the control problem. Their actions are Acceleration, Deceleration, Change lane to the left, Change lane to the right, and Take no action, where the first two apply maximal acceleration and deceleration, while the two lanechanging actions simply use constant speed lateral movements.…”

Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles

“…In the area of motion planning, the end episode rewards are calculated from the fulfillment or failure of the driving task. The overall performance factors are generally: time of finishing the task, keeping the desired speed or achieving as high average speed as possible, yaw or distance from lane middle or the desired trajectory, overtaking more vehicles, achieve as few lane changes as possible [44], keeping right [45], [46] etc. Rewarding systems also can represent passenger comfort, where the smoothness of the vehicle dynamics is enforced.…”

Evaluation Model of Autonomous Vehicles’ Speed Suitability Based on Overtaking Frequency

“…Zhu et al proposed a human-like autonomous car following a planning framework based on deep reinforcement learning and established a human-like car-following model [ 22 ]. In addition, some driving decisions based on learning algorithms have been extensively studied in recent researches [ 23 , 24 ].…”