On Multi-Agent Reinforcement Learning in Matrix, Stochastic and Differential Games

Abstract

In this thesis, we investigate reinforcement learning algorithms on matrix, stochastic, and differential games. In matrix and stochastic games, the states and actions are represented in continuous domains. We propose two decentralized multi-agent reinforcement learning algorithms to solve the problem of learning in matrix and stochastic games when the learning agent has only minimum knowledge about the underlying game and the other learning agents. The proposed algorithms are the constant learning rate-based exponential moving average Q-learning (CLR-EMAQL) algorithm, and the exponential moving average Q-learning (EMAQL) algorithm. We mathematically show that the proposed CLR-EMAQL algorithm converges to Nash equilibrium in games with pure Nash equilibrium. We introduce the concept of Win-or-Learn-Slow (WoLS) mechanism for the proposed EMAQL algorithm so that the proposed algorithm learns fast when it is winning, and learns cautiously when it is losing. We also provide a theoretical proof of convergence to Nash equilibrium for the proposed EMAQL algorithm in games with pure Nash equilibrium. In games with mixed Nash equilibrium, our mathematical analysis shows that the proposed EMAQL algorithm converges to an equilibrium. Although our mathematical analysis does not explicitly show that the proposed EMAQL algorithm converges to Nash equilibrium, our simulation results indicate that the proposed EMAQL algorithm does converge to Nash equilibrium. Our simulation iii I would like to express my deepest gratitude to my advisor, Professor Howard M.

Fictitious Play in Zero-Sum Stochastic Games

“…For example, in [6] and more recently [2], the authors focused on scenarios where players play in a coordinated manner a finite-horizon version of a zero-sum stochastic game within repeated episodes, referred to as episodic reinforcement learning, even though player payoffs are defined over an infinite horizon. In another line of work, in [32] and [1], the authors presented and studied algorithms that update policies only at certain time instances while keeping them fixed in between-even when players may have an incentive to change their actions-in order to create a stationary environment for learning the underlying model or estimating the associated Q-functions.…”