Linear Quadratic Control Using Model-Free Reinforcement Learning

Abstract

In this paper, we consider Linear Quadratic (LQ) control problem with process and measurement noises. We analyze the LQ problem in terms of the average cost and the structure of the value function. We assume that the dynamics of the linear system is unknown and only noisy measurements of the state variable are available. Using noisy measurements of the state variable, we propose two model-free iterative algorithms to solve the LQ problem. The proposed algorithms are variants of policy iteration routine where the policy is greedy with respect to the average of all previous iterations. We rigorously analyze the properties of the proposed algorithms, including stability of the generated controllers and convergence. We analyze the effect of measurement noise on the performance of the proposed algorithms, the classical off-policy, and the classical Q-learning routines. We also propose a model-building approach where a model of the dynamical system is estimated and the optimal control problem is solved assuming that the estimated model is the true model. We use a benchmark to evaluate and compare our proposed algorithms with the classical off-policy, the classical Q-learning, and the policy gradient. We show that our model-building approach performs nearly identical to the analytical solution and our proposed policy iterationbased algorithms outperform the classical off-policy and the classical Q-learning algorithms on this benchmark but do not outperform the model-building approach.

Quadratic Tracking Control of Linear Stochastic Systems with Unknown Dynamics Using Average Off-Policy Q-Learning Method

“…In order to better explain Algorithm 2, we can divide it into the following three steps: First of all, prove that Ĝi can be uniquely determined in each iteration. Lemma 5 explains that the solutions obtained from the Bellman Equation ( 26) based on the average off-policy Q-learning method are equivalent to those obtained from the model-based Bellman Equation (18). Equation (46) of Algorithm 2 is derived from the transformation of Equation ( 26), and we show that the procedure in Algorithm 2 guarantees the existence of the solution Ĝi .…”

“…By utilizing an average cost function, Reference [17] tackled the output regulation problem for linear systems with unknown dynamics. Data-driven average Q-learning algorithms can also handle linear quadratic control problems, especially when there are un-measurable stochastic disturbances [18,19].…”

“…However, that paper [25] did not consider external noises in the system. In reference [18], the authors considered the case of external noises to address the optimal control problem (OCP), but the tracking control problem was not taken into account. In reference [26], the Q-learning method was employed to address the LQT problem in the presence of partially unknown system dynamics.…”

A deep learning optimized LQR method for enhanced formation control with embedded systems

“…An iterative Linear Quadratic Regulator (iLQR) optimal control technique based on the dynamic modeling of the quadrotors was developed to achieve the leader-follower formation (Jasim and Gu 2019). Among the array of control methodologies discussed earlier, Reinforcement Learning (RL)-based control approaches offer an end-to-end solution for motion control (Yaghmaie et al 2023) and obstacle avoidance (Sadhukhan and Selmic 2021) during formation.…”

Trajectory tracking for quadrotors: An optimization‐based planning followed by controlling approach

“…However, adding a terminal constraints set increases computational demand at each step. Thus, we proposed a LQG-based approach (Yaghmaie et al, 2022) to reduce the disturbance and increase system stability.…”