Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles

Abstract

Academic research in the field of autonomous vehicles has reached high popularity in recent years related to several topics as sensor technologies, V2X communications, safety, security, decision making, control, and even legal and standardization rules. Besides classic control design approaches, Artificial Intelligence and Machine Learning methods are present in almost all of these fields. Another part of research focuses on different layers of Motion Planning, such as strategic decisions, trajectory planning, and control. A wide range of techniques in Machine Learning itself have been developed, and this article describes one of these fields, Deep Reinforcement Learning (DRL).The paper provides insight into the hierarchical motion planning problem and describes the basics of DRL. The main elements of designing such a system are the modeling of the environment, the modeling abstractions, the description of the state and the perception models, the appropriate rewarding, and the realization of the underlying neural network. The paper describes vehicle models, simulation possibilities and computational requirements. Strategic decisions on different layers and the observation models, e.g., continuous and discrete state representations, grid-based, and camera-based solutions are presented. The paper surveys the state-of-art solutions systematized by the different tasks and levels of autonomous driving, such as car-following, lane-keeping, trajectory following, merging, or driving in dense traffic. Finally, open questions and future challenges are discussed.

Attributation Analysis of Reinforcement Learning-Based Highway Driver

“…Such an approach allows for the increased control and understanding of the algorithm. A more detailed overview of the works on the application of RL in motion planning can be found in [12,13].…”

Combining Reinforcement Learning with Model Predictive Control for On-Ramp Merging

“…On the other hand RL-based techniques have only recently gained traction in the domain of self-driving, primarily due to their recent astonishing success in various other domains (see [7], [8] and [9]). Despite being relatively recent, RL-based techniques have been investigated for both motion planning in general (see [10] for a survey) and on-ramp merging in particular (see [1], [2] and [11]). As we will demonstrate in our simulations section, since RL agents are trained using copious amounts of data to maximize some notion of longterm reward (which is typically a function of passenger comfort, safety and efficiency, defined later in the paper) and do not just focus on safety, their safety performance is subpar.…”

“…Some other popular RL algorithms such as DQN require discrete action spaces, and had worse performance than DDPG in our experiments. DDPG has been widely used to solve autonomous driving tasks ( [10], [19], [1]), including on-ramp merging. For the on-ramp merging problem, our approach is most similar to that found in [2], as they use a similar highway, DDPG, and also aim to minimize jerk.…”

An End-to-end Deep Reinforcement Learning Approach for the Long-term Short-term Planning on the Frenet Space

“…DRL's has shown a satisfactory performance on behavior planning. Nevertheless, as pointed out in the recent survey paper [17], few studies applied DRL on motion planning for autonomous driving systems. Feher et al [18] trained a DDPG agent to generate the waypoints for the vehicle to track.…”