Reinforcement Learning-Based Motion Planning for Automatic Parking System

Abstract

In automatic parking motion planning, multi-objective optimization including safety, comfort, parking efficiency, and final parking performance should be considered. Most of the current research relies on the parking data from expert drivers or prior knowledge of humans. However, it is challenging to obtain a large amount of high-quality expert drivers' data. Furthermore, expert drivers' data or prior knowledge of humans does not guarantee an optimal multi-objective parking performance. In this paper, we propose a model-based reinforcement learning method that learns parking policy of the data, by executing the data generation, data evaluation, and training network, iteratively. The trained network is used to guide the data generation cycle in the subsequent iteration. Based on this proposed method, we can get rid of human experience largely and learn parking strategies autonomously and quickly. The learned strategies ensure the multi-objective optimality of above requirements in the parking process. First, an environment model that approximates the actual environment is established, and the learning efficiency is accelerated through the simulated interaction between the agent and the environment model. To make the system independent of expert data or prior knowledge, a data generation algorithm combining Monte Carlo Tree Search (MCTS) and longitudinal and lateral policies is proposed. Then, to meet the multi-objective optimal demands mentioned above, a reward function is constructed to evaluate and filter the parking data. Finally, a neural network is used to learn the parking strategy from the filtered data. From the real vehicle test benchmarked with a mass-produced parking system, the proposed method is found to achieve better parking efficiency and lower requirements for start parking posture, thereby verifying the algorithm's superiority.

Data Efficient Reinforcement Learning for Integrated Lateral Planning and Control in Automated Parking System

“…Data inefficiency makes model-free RL impractical and limits its application in parking scenarios, in which the vehicle is required to quickly acquire driving skills. Model-based reinforcement learning [ 10 ] was used to realize multi-objective optimization and get rid of human experience. However, as the action was determined by the results of simulations, this method heavily relied on the accuracy of the vehicle model.…”

“…Little research has been conducted to address the data efficiency for RL-based APS especially for model-based RL APS or other automatic driving scenes while maintaining the continuous learning ability. Regarding model-based RL technology, AlphaGo [ 12 ] utilized the basic rules in the game of Go and function approximation to get the state value function and to defeat the human player by self-play, which inspired the combining of [ 10 ] and the state value function to overcome the shortcoming of model-based RL parking system.…”

“…Designing a reward function for the final performance is simpler and more objective than offering immediate rewards. Unlike the model-based method used in [ 10 ], the requirement for the accuracy of the vehicle model can be reduced by limiting the use of simulation in the search process. MCTS uses the estimation of the state value and the probability distribution of candidate actions.…”

“…MCTS uses the estimation of the state value and the probability distribution of candidate actions. In addition to the one artificial neural network (ANN) used to learn the probability distribution of the action with high reward [ 10 ], another value ANN is used to predict the state value. To further improve the data efficiency, several methods are proposed.…”

“…The vehicle model is used to generate imagination rollouts, which is supplemented with real parking data. The reasons for using DERL are threefold: First, the Monte Carlo structure is suitable for parking tasks [ 10 ], where rewards are sparse. Second, the use of the value ANN can truncate the search process and avoid the accumulation of errors.…”