Data Efficient Reinforcement Learning for Integrated Lateral Planning and Control in Automated Parking System

Abstract

Reinforcement learning (RL) is a promising direction in automated parking systems (APSs), as integrating planning and tracking control using RL can potentially maximize the overall performance. However, commonly used model-free RL requires many interactions to achieve acceptable performance, and model-based RL in APS cannot continuously learn. In this paper, a data-efficient RL method is constructed to learn from data by use of a model-based method. The proposed method uses a truncated Monte Carlo tree search to evaluate parking states and select moves. Two artificial neural networks are trained to provide the search probability of each tree branch and the final reward for each state using self-trained data. The data efficiency is enhanced by weighting exploration with parking trajectory returns, an adaptive exploration scheme, and experience augmentation with imaginary rollouts. Without human demonstrations, a novel training pipeline is also used to train the initial action guidance network and the state value network. Compared with path planning and path-following methods, the proposed integrated method can flexibly co-ordinate the longitudinal and lateral motion to park a smaller parking space in one maneuver. Its adaptability to changes in the vehicle model is verified by joint Carsim and MATLAB simulation, demonstrating that the algorithm converges within a few iterations. Finally, experiments using a real vehicle platform are used to further verify the effectiveness of the proposed method. Compared with obtaining rewards using simulation, the proposed method achieves a better final parking attitude and success rate.

Model Predictive Control-Based Integrated Path Tracking and Velocity Control for Autonomous Vehicle with Four-Wheel Independent Steering and Driving

“…The constraints for the state vector of the MPC problem are defined as (25). The input constraints are defined as (26) and (27), which mean a maximum value and jerk limit to each actuator needed to consider the actuator limit. The inequality constraints on steering of 4WIS are defined as (26).…”

“…In (26), δ max and δ' max are the maximum steering angle and its angular rate, respectively. The inequality constraints for drive torque of 4WID are defined as (27). In (27), F x,max, and F' x,max are the maximum longitudinal tire force and its rate, respectively.…”

“…The inequality constraints for drive torque of 4WID are defined as (27). In (27), F x,max, and F' x,max are the maximum longitudinal tire force and its rate, respectively. Finally, the parameters of the linear time-varying MPC are summarized in Table 1.…”

“…Moreover, a multi-layered MPC was developed to compensate for the modeling error by using three different models [25]. Recently, a learning-based steering control algorithm were proposed to achieve acceptable performance without modeling [26][27][28]. For velocity tracking control, MPC with V2V communication was adopted to reduce the rear-end collision [29].…”

Fast Motion Model of Road Vehicles with Artificial Neural Networks

“…Similarly, authors of [23] combine Model Predictive Control (MPC) with RBFNN to robustly handle the nonlinear characteristics of the steering system. A reinforcement-learning-based integrated planning and control method is proposed for automated parking applications in [24], which can simultaneously coordinate the longitudinal and lateral motions to park in a smaller parking space in one maneuver. Reinforcement learning is also used for high velocity lane change maneuvering in [25], where a Deep Deterministic Policy Gradient (DDPG) agent is utilized in an end-to-end method using lidar data.…”