Reinforcement Learning Approach to Design Practical Adaptive Control for a Small-Scale Intelligent Vehicle

Abstract

Reinforcement learning (RL) based techniques have been employed for the tracking and adaptive cruise control of a small-scale vehicle with the aim to transfer the obtained knowledge to a full-scale intelligent vehicle in the near future. Unlike most other control techniques, the purpose of this study is to seek a practical method that enables the vehicle, in the real environment and in real time, to learn the control behavior on its own while adapting to the changing circumstances. In this context, it is necessary to design an algorithm that symmetrically considers both time efficiency and accuracy. Meanwhile, in order to realize adaptive cruise control specifically, a set of symmetrical control actions consisting of steering angle and vehicle speed needs to be optimized simultaneously. In this paper, firstly, the experimental setup of the small-scale intelligent vehicle is introduced. Subsequently, three model-free RL algorithm are conducted to develop and finally form the strategy to keep the vehicle within its lanes at constant and top velocity. Furthermore, a model-based RL strategy is compared that incorporates learning from real experience and planning from simulated experience. Finally, a Q-learning based adaptive cruise control strategy is intermixed to the existing tracking control architecture to allow the vehicle slow-down in the curve and accelerate on straightaways. The experimental results show that the Q-learning and Sarsa (λ) algorithms can achieve a better tracking behavior than the conventional Sarsa, and Q-learning outperform Sarsa (λ) in terms of computational complexity. The Dyna-Q method performs similarly with the Sarsa (λ) algorithms, but with a significant reduction of computational time. Compared with a fine-tuned proportion integration differentiation (PID) controller, the good-balanced Q-learning is seen to perform better and it can also be easily applied to control problems with over one control actions.

Adaptive Human–Machine Evaluation Framework Using Stochastic Gradient Descent-Based Reinforcement Learning for Dynamic Competing Network

“…In addition to robot control, RL is applied and used effectively in various fields. For convenience in daily life, RL has been applied to drone delivery, home energy system optimization, autonomous driving, and automatic parking systems [12][13][14][15]. In Internet of Things devices and networks, RL is mainly used to control traffic and congestion in complex situations.…”

A Study on an Enhanced Autonomous Driving Simulation Model Based on Reinforcement Learning Using a Collision Prevention Model

“…It requires high technological development in that driver intervention is unnecessary in all driving situations and the vehicle can be driven without a driver. There is an absolute need for image processing [2][3] and AI in order to realize completely autonomous driving [4][5][6][7][8]. These technologies comprise the core of autonomous driving, across all of its stages.…”

“…It is not general practice to embody all functions of autonomous driving only on the basis of reinforcement learning. Research has been conducted to create local routes, predict collisions, and brake for autonomous driving vehicles on the basis of reinforcement There is an absolute need for image processing [2,3] and AI in order to realize completely autonomous driving [4][5][6][7][8]. These technologies comprise the core of autonomous driving, across all of its stages.…”

A Waypoint Tracking Controller for Autonomous Road Vehicles Using ROS Framework

“…Dynamic controllers [33][34][35] include the dynamic properties of the vehicles in the control law, but need dynamic feedbacks such as force and torque which require expensive dedicated sensors. When high robustness to tracking disturbances is required, a good choice is using adaptive and intelligent control [36,37], although these methods usually take a lot of computational effort. On the other extreme, classical controllers [38,39] can be easily applied to steering actuation, but their adjustment may need complex derivations and selections.…”