Experience replay–based output feedback Q‐learning scheme for optimal output tracking control of discrete‐time linear systems

Abstract

This paper focuses on solving the adaptive optimal tracking control problem for discrete-time linear systems with unknown system dynamics using output feedback. A Q-learning-based optimal adaptive control scheme is presented to learn the feedback and feedforward control parameters of the optimal tracking control law. The optimal feedback parameters are learned using the proposed output feedback Q-learning Bellman equation, whereas the estimation of the optimal feedforward control parameters is achieved using an adaptive algorithm that guarantees convergence to zero of the tracking error. The proposed method has the advantage that it is not affected by the exploration noise bias problem and does not require a discounting factor, relieving the two bottlenecks in the past works in achieving stability guarantee and optimal asymptotic tracking. Furthermore, the proposed scheme employs the experience replay technique for data-driven learning, which is data efficient and relaxes the persistence of excitation requirement in learning the feedback control parameters. It is shown that the learned feedback control parameters converge to the optimal solution of the Riccati equation and the feedforward control parameters converge to the solution of the Sylvester equation. Simulation studies on two practical systems have been carried out to show the effectiveness of the proposed scheme.

The Adaptive Optimal Output Feedback Tracking Control of Unknown Discrete-Time Linear Systems Using a Multistep Q-Learning Approach

“…In this paper, we propose a tracking control approach that utilizes a static output feedback multistep Q-learning algorithm in conjunction with state reconstruction techniques. A separate adaptation mechanism was introduced in [38] to estimate unknown feedforward tracking terms. However, the static OPFB design we propose owes its popularity to its simplicity in terms of structure.…”

Optimal control of a two‐wheeled self‐balancing robot by reinforcement learning

“…Recently, reinforcement learning has emerged as an effective method to solve optimal control problems without using system model information 8‐10 . Q‐learning is a type of reinforcement learning algorithm, which is completely online in nature and does not use any prior information of system dynamics.…”

H∞ tracking control for linear discrete‐time systems via reinforcement learning

“…In recent decades, a large number of researches on the problem of H ∞ control have emerged . Meanwhile, the tracking control problem has been widely studied as it appears in many practical control applications . Combining tracking control with H ∞ control, it can attenuate the effects of disturbances on system and ensure the system trajectory tracks the reference signal.…”

“…[3][4][5][6] Meanwhile, the tracking control problem has been widely studied as it appears in many practical control applications. [7][8][9] Combining tracking control with H ∞ control, it can attenuate the effects of disturbances on system and ensure the system trajectory tracks the reference signal. Therefore, the research on H ∞ tracking control is meaningful.…”