No abstract

Reinforcement learning-based approaches are widely studied in the literature for solving different control tasks for Connected and Autonomous Vehicles, from which this paper deals with the problem of lateral control of a dynamic nonlinear vehicle model, performing the task of lane-keeping. In this area, the appropriate formulation of the goals and environment information is crucial, for which the research outlines the importance of lookahead information, enabling to accomplish maneuvers with complex trajectories. Another critical part is the real-time manner of the problem. On the one hand, optimization or search based methods, such as the presented Monte Carlo Tree Search method, can solve the problem with the trade-off of high numerical complexity. On the other hand, single Reinforcement Learning agents struggle to learn these tasks with high performance, though they have the advantage that after the training process, they can operate in a real-time manner. Two planning agent structures are proposed in the paper to resolve this duality, where the machine learning agents aid the tree search algorithm. As a result, the combined solution provides high performance and low computational needs.

The paper presents a motion planning solution which combines classic control techniques with machine learning. For this task, a reinforcement learning environment has been created, where the quality of the fulfilment of the designed path by a classic control loop provides the reward function. System dynamics is described by a nonlinear planar single track vehicle model with dynamic wheel mode model. The goodness of the planned trajectory is evaluated by driving the vehicle along the track. The paper shows that this encapsulated problem and environment provides a one-step reinforcement learning task with continuous actions that can be handled with Deep Deterministic Policy Gradient learning agent. The solution of the problem provides a real-time neural network-based motion planner along with a tracking algorithm, and since the trained network provides a preliminary estimate on the expected reward of the current state-action pair, the system acts as a trajectory feasibility estimator as well.

Over the last decade, many different algorithms were developed for the motion planning of road vehicles due to the increasing interest in the automation of road transportation. To be able to ensure dynamical feasibility of the planned trajectories, nonholonomic dynamics of wheeled vehicles must be considered. Nonlinear optimization based trajectory planners are proven to satisfy this need, however this happens at the expense of increasing computational effort, which jeopardizes the real-time applicability of these methods. This paper presents an algorithm which offers a solution to this problematic with a hybrid approach using artificial neural networks (ANNs). First, a nonlinear optimization based trajectory planner is presented which ensures the dynamical feasibility with the model-based prediction of the vehicle's motion. Next, an artificial neural network is trained to reproduce the behavior of the optimization based planning algorithm with the method of supervised learning. The generation of training data happens off-line, which eliminates the concerns about the computational requirements of the optimization-based method. The trained neural network then replaces the original motion planner in on-line planning tasks which significantly reduces computational effort and thus run-time. Furthermore, the output of the network is supervised by the model based motion prediction layer of the original optimization-based algorithm and can thus always be trusted. Finally, the performance of the hybrid method is benchmarked with computer simulations in terms of dynamical feasibility and run-time and the results are investigated. Examinations show that the computation time can be significantly reduced while maintaining the feasibility of resulting vehicle motions.

Nonlinear optimization-based motion planning algorithms have been successfully used for dynamically feasible trajectory planning of road vehicles. However, the main drawback of these methods is their significant computational effort and thus high runtime, which makes real-time application a complex problem. Addressing this field, this paper proposes an algorithm for fast simulation of road vehicle motion based on artificial neural networks that can be used in optimization-based trajectory planners. The neural networks are trained with supervised learning techniques to predict the future state of the vehicle based on its current state and driving inputs. Learning data is provided for a wide variety of randomly generated driving scenarios by simulation of a dynamic vehicle model. The realistic random driving maneuvers are created on the basis of piecewise linear travel velocity and road curvature profiles that are used for the planning of public roads. The trained neural networks are then used in a feedback loop with several variables being calculated by additional numerical integration to provide all the outputs of the original dynamic model. The presented model can be capable of short-term vehicle motion simulation with sufficient precision while having a considerably faster runtime than the original dynamic model.