Virtual to Real Reinforcement Learning for Autonomous Driving

Abstract

Reinforcement learning is considered as a promising direction for driving policy learning. However, training autonomous driving vehicle with reinforcement learning in real environment involves non-affordable trial-and-error. It is more desirable to first train in a virtual environment and then transfer to the real environment. In this paper, we propose a novel realistic translation network to make model trained in virtual environment be workable in real world. The proposed network can convert non-realistic virtual image input into a realistic one with similar scene structure. Given realistic frames as input, driving policy trained by reinforcement learning can nicely adapt to real world driving. Experiments show that our proposed virtual to real (VR) reinforcement learning (RL) works pretty well. To our knowledge, this is the first successful case of driving policy trained by reinforcement learning that can adapt to real world driving data.Autonomous driving aims to make a vehicle sense its environment and navigate without human input. To achieve this goal, the most important task is to learn the driving policy that automatically outputs control signals for steering wheel, throttle, brake, etc., based on observed surroundings. The straight-forward idea is end-to-end supervised learning [3,4], which trains a neural network model mapping visual input directly to action output, and the training data is labeled image-action pairs. However, supervised approach usually requires large amount of data to train a model [31] that can generalize to different environments. Obtaining such amount of data is time consuming and requires significant human involvement. By contrast, reinforcement learning learns by a trial-and-error fashion, and does not require explicit supervision from human. Recently, reinforcement learning has been considered as a promising technique to learn driving policy due to its expertise in action planing [15,23,25].However, reinforcement learning requires agents to interact with environments, and undesirable driving actions would happen. Training autonomous driving cars in real world will cause damages to vehicles and the surroundings. Therefore, most of current research in autonomous driving with reinforcement learning focus on simulations [15,18,25] rather than training in real world. While an agent trained with reinforcement learning achieves

A Survey of Deep Learning Applications to Autonomous Vehicle Control

“…The agent then processes these inputs through an RNN to provide a heading, speed, and waypoint, which are then achieved through a low-level controller. This had the advantage that pre-processed inputs could be obtained either from simulation or real-world data, which makes transferring driving policies from simulation to the real world easier [143], [144]. Furthermore, synthesising perturbations to model recoveries from incorrect lane positions or even scenarios such as collisions or driving off-road provides the model with robustness to errors and allows the model to learn to avoid such scenarios.…”

A Survey on Visual Traffic Simulation: Models, Evaluations, and Applications in Autonomous Driving

“…Pan et al . [PYWL17] use a novel realistic translation network to train an autonomous driving model in a virtual environment and then use it in the real‐world environment. In this virtual‐to‐real reinforcement learning framework, the images from virtual environment are segmented to scene‐parsing representations first and then are translated to synthetic images.…”

Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehicles

“…Conversely, discretization is avoided in [9], by introducing a continuous non-linear optimization method where obstacles in the form of polygons are converted to quadratic constraints. c) Learned Motion Planning: Learning approaches to motion planning have mainly been studied from an imitation learning (IL) [10], [11], [12], [13], [14], [15], [16], or reinforcement learning (RL) [17], [18], [19], [20] perspective. While most IL approaches provide an end-to-end training framework to control outputs from sensory data, they suffer from compounding errors due to the sequential decision making process of self-driving.…”