H_∞ Control for Discrete-Time Multi-Player Systems via Off-Policy Q-Learning

Abstract

This paper presents a novel off-policy game Q-learning algorithm to solve H ∞ control problem for discrete-time linear multi-player systems with completely unknown system dynamics. The primary contribution of this paper lies in that the Q-learning strategy employed in the proposed algorithm is implemented in an off-policy policy iteration approach other than on-policy learning, since the off-policy learning has some well-known advantages over the on-policy learning. All of players struggle together to minimize their common performance index meanwhile defeating the disturbance that tries to maximize the specific performance index, and finally they reach the Nash equilibrium of game resulting in satisfying disturbance attenuation condition. For finding the solution of the Nash equilibrium, H ∞ control problem is first transformed into an optimal control problem. Then an off-policy Q-learning algorithm is put forward in the typical adaptive dynamic programming (ADP) and game architecture, such that control policies of all players can be learned using only measured data. More importantly, the rigorous proof of no bias of solution to the Nash equilibrium by using the proposed off-policy game Q-learning algorithm is presented. Comparative simulation results are provided to verify the effectiveness and demonstrate the advantages of the proposed method. INDEX TERMS H ∞ control, off-policy Q-learning, game theory, Nash equilibrium.

Output Feedback H∞ Control for Linear Discrete-Time Multi-Player Systems With Multi-Source Disturbances Using Off-Policy Q-Learning

“…Compared (41) and (42) with (22) and (23) derived in this paper for solving output feedback H ∞ control of DT systems, one can easily find that solving (22) and (23) seems to be more difficult than (41) and (42), since no coupling relationship of control policy gains and disturbance policy gains is in (41) and (42), which is caused by the natural difference of DT systems from CT systems.…”

“…Theorem 1 is used to prove the uniqueness of the control policies and the disturbances in (23). Theorem 1: Given Assumption 2, then there is a unique matrixH that makes (22) hold, and the unique optimal control policies and the worst disturbances will be solved by (23). Moreover, (23) is the same as (16).…”

“…The initial controller gains K 0 i and disturbance gains K 0 d l , and as well as the behavior control policies and behavior disturbances can be chosen empirically or utilize the robust control method like [2] and [20]. Remark 5: It is worth pointing out that Algorithm 2 presents a completely model-free approach to fix out the output feedback H ∞ control problem by finding the Nash equilibrium solution using off-policy Q-learning combined with game theory, which is the first time attempt for multiplayer systems resisting multi-source disturbances under the foundation laid by [16] with H 2 /H ∞ state feedback control using model-based approach, [24]- [26] with data-driven RL algorithms for CT systems, and [23] with RL algorithm worked for state-feedback controller design of multi-player systems with single-source disturbance. Moreover, we extend the H ∞ output feedback controller design [27]- [29] for single-controller systems into the case of multi-player and multi-source disturbances systems by developing a novel off-policy MCC game Q-learning algorithm.…”

“…Notice that the above-mentioned reports [17]- [20], [22], [23] can work for rejecting only one disturbance source or compacted augmented disturbance from multiple sources. Thus, one issue is naturally raised, that is how to deal with anti-interference of multi-source disturbances of multiplayer systems by intelligently interacting with each other of all players and all kinds of disturbances, since multisource disturbances, such as time-varying temperature, raw material, price, inevitably exist in practice, such as power systems, large-scale industrial processes, circuit systems [11], [13], [15].…”

Robust Control of Unknown Observable Nonlinear Systems Solved as a Zero-Sum Game

“…In [35], a data-based policy iteration Q-learning algorithm for ZS-TP-G was developed for linear systems to eliminate process dynamics knowledge. Recent game-theoretical contributions (some in nonzero-sum games) for nonlinear systems are reported in [36]- [38], in the more general framework of robust control [39]- [40].…”