Learning near‐optimal broadcasting intervals in decentralized multi‐agent systems using online least‐square policy iteration

Abstract

Here, agents learn how often to exchange information with neighbours in cooperative multi-agent systems (MASs) such that their linear quadratic regulator (LQR)-like performance indices are minimized. The investigated LQR-like cost functions capture trade-offs between the energy consumption of each agent and MAS local control performance in the presence of exogenous disturbances, delayed and noisy data. Agent energy consumption is critical for prolonging the MAS mission and is composed of both control (e.g. acceleration, velocity) and communication efforts. Taking provably stabilizing upper bounds on broadcasting intervals as optimization constraints, an online off-policy model-free learning algorithm based on least square policy iteration (LSPI) to minimize the cost function of each agent is employed. Consequently, the obtained broadcasting intervals adapt to the most recent information (e.g. delayed and noisy agents' inputs and/or outputs) received from neighbours whilst provably stabilize the MAS. Chebyshev polynomials are utilized as the approximator in the LSPI whereas Kalman filtering handles sampled, corrupted, and delayed data. Subsequently, convergence and near-optimality of our LSPI scheme are inspected. The proposed methodology is verified experimentally using an inexpensive motion capture system and nano quadrotors.

Extending the LQR to the design of PID type‐ii and type‐iii control loops

“…Modern optimal control theory has a rich set of analytical tools to design control strategies satisfying desirable characteristics of the excursion of the system states according to the designer's specifications in an optimal manner [19]. The LQR is one such design methodology whereby quadratic performance indices involving the control signal and the state variables are minimized in an optimal fashion [20][21][22][23]. It has a very nice stability property, that is, if the process is of single-input and single-output, then the control system has at least the phase margin of 60 • and the gain margin of infinity [24].…”