LQ control of unknown discrete‐time linear systems—A novel approach and a comparison study

Abstract

In this paper, we propose a novel approach to the linear quadratic (LQ) optimal control of unknown discrete-time linear systems. We first describe an iterative procedure for minimizing a partially unknown static function. The procedure is based on simultaneous updates in the estimation of unknown parameters and in the optimization of controllable inputs. We then use the procedure for control optimization in unknown discrete-time dynamic systems-we consider applications to the finite-horizon and the infinite-horizon LQ control of linear systems in detail. To illustrate the approach, an example of the pitch attitude control of an aircraft is considered. We also compare our proposed approach to several other approaches to finite/infinite-horizon LQ control problems with unknown dynamics from the literature, including extremum seeking and adaptive dynamic programming/reinforcement learning. Our proposed approach is competitive with these approaches in speed of convergence and in implementation and computational complexity. KEYWORDSLQ control, optimal control, unknown system Optim Control Appl Meth. 2019;40:265-291. wileyonlinelibrary.com/journal/oca © 2018 John Wiley & Sons, Ltd. 265 266 LI ET AL.To cope with the lack of accurate models or the changes in dynamics, control techniques such as adaptive control 3 and robust optimal control 4 can be applied. The goal of adaptive control is, in principle, different from that of optimal control. On the one hand, adaptive control usually concerns itself with the adaptation of a fixed-form controller to initially uncertain system parameters to achieve typical control requirements such as stabilization and reference tracking. On the other hand, the goal of optimal control is to find a control that minimizes a specified cost function, while stabilization and reference tracking may be achieved through the cost function design. Robust optimal control pursues such a goal for systems with uncertain parameters: it optimizes the control to minimize the worst cost (in a min-max formulation) 5,6 or the cost expectation (in a probability-weighted-average formulation) 7 over an uncertainty set.An alternative technique is optimal control for unknown systems, which is also referred to as model-free optimal control, ie, optimal control without the need for prior knowledge of the system dynamics. Differently from the problem setting of robust optimal control, model-free optimal control typically pursues control optimization to minimize the true cost associated with a given but unknown system.Model-free optimal control approaches relying on learning and dynamic programming have been developed, for example, in the works of Dierks et al, 8 Lewis and Vamvoudakis, 9 and Wang et al. 10 In the work of Dierks et al, 8 a neural network is used to learn the plant dynamics online; then, an adaptive dynamic programming (ADP) algorithm is exploited to obtain an optimal control law off-line based on the learned neural network model. In the work of Lewis and Vamvoudakis, 9 policy iteration and value itera...

Model‐free optimal tracking over finite horizon using adaptive dynamic programming

“…. Now, it can be shown that there exists a finite horizon admissible control of length 2 so that corresponding cumulative cost V (2) (e k ) is related to V (1) (e k ). To that end, we construct finite horizon admissible control of length 2:…”

“…Optimal control for nonlinear systems requires solving the nonlinear Hamilton-Jacobi-Bellman (HJB) equation analytically which is usually too difficult. 1,2 To that end, the usefulness of dynamic programming (DP) for solving HJB equation is well established. [3][4][5] Adaptive dynamic programming (ADP) based methods have emerged as effective approach for solving nonlinear HJB in approximative sense and solve the optimal control problems forward-in-time.…”

A New Algorithm for the LQR Problem with Partially Unknown Dynamics

“…We treat each row r k of A separately. Denoting by y j the right hand side in (7), we can write its k-th component as…”

“…Finally, let us mention that other works have already dealt with LQR problems with unknown dynamics (see i.e. [4,5,7,9] and references therein), but they all need several trials to converge, whereas our method works with just one.…”

A New Algorithm for the LQR Problem with Partially Unknown Dynamics

“…Finally, let us mention that other works already dealt with LQR problems with unknown dynamics (see i.e. [4,5,7,9] and references therein), but they all need several trials to converge, whereas our method works with just one simulation.…”