Output‐feedback Q‐learning for discrete‐time linear H^∞ tracking control: A Stackelberg game approach

Abstract

In this article, an output‐feedback Q‐learning algorithm is proposed for the discrete‐time linear system to deal with the H∞$$ {H}_{\infty } $$ tracking control problem. The problem is formulated as a zero‐sum game in the Stackelberg game framework with a discount factor to make the value function bounded. According to the principle of optimality, the game algebraic Riccati equation (GARE) is derived and solved by the Q‐learning algorithm to get the optimal solution of the Stackelberg game without requiring the knowledge of system dynamics and state. It is proved that the solution of the algorithm converges to the optimal control input and the worst‐case disturbance with excitation noises during training, and the Stackelberg strategy can achieve a lower L2$$ {L}_2 $$ disturbance attenuation level than the Nash one. Moreover, the impacts of the discount factor on the stability of the closed‐loop system and solvability of the GARE are analyzed to provide some criteria for the choice of the discount factor. Simulation examples are provided to validate the effectiveness of the algorithm.

Preview controller synthesis for a class of linear parameter periodic systems

“…For example, reference 8 studied the output tracking control problem of Hamiltonian descriptor systems with input saturation and delay. Moreover, reference 9 developed an output‐feedback Q‐learning algorithm for discrete‐time linear systems to solve the problem of tracking control.…”

Finite‐horizon H∞ tracking control for discrete‐time linear systems

“…[24][25][26] Q-learning algorithm has also been proposed to solve the game algebraic Riccati equation (GARE) of discrete-time linear systems. [27][28][29][30][31] The solution of the GARE leads to the realization of H ∞ tracking control.…”

“…By combining the Q‐function, RL methods can achieve model‐free optimal control of CT or discrete‐time (DT) systems 24‐26 . Q‐learning algorithm has also been proposed to solve the game algebraic Riccati equation (GARE) of discrete‐time linear systems 27‐31 . The solution of the GARE leads to the realization of $$$ {H}_{\infty } $$$ tracking control.…”

“…However, the above‐mentioned RL‐based $$$ {H}_{\infty } $$$ tracking control methods 15‐21,27‐31 concern the infinite‐horizon cases. These methods can only achieve asymptotic $$$ {H}_{\infty } $$$ tracking control, which means that the theoretical regulation time tends to be infinite.…”

“…The innovation of this paper can be given by comparing it with the existing works. Firstly, the discount factor which plays a key role in infinite‐horizon $$$ {H}_{\infty } $$$ tracking controller design 15‐21,27‐31 is not required in the developed finite‐horizon $$$ {H}_{\infty } $$$ tracking control method. The discount factor will affect the tracking error, whereas the developed method in this paper will not encounter this problem.…”