Q-Learning in Regularized Mean-field Games

Anahtarcı, Berkay; Karıksız, Can Deha; Saldi, Naci

doi:10.1007/s13235-022-00450-2

Cited by 18 publications

(6 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence assumption (d) is not true for this new formulation. To handle this problem, a common approach is to add a strongly convex regularization term to the cost function cnew (see [3]). In regularized version, the cost function is given by creg (x, u, z) := cnew (x, u, z) + λ h(u) where h is a θ-strongly convex function and λ > 0 is some constant.…”

Section: Computing Equilibrium Of Gnepmentioning

confidence: 99%

Linear Mean-Field Games with Discounted Cost

Saldi¹

2023

Preprint

View full text Add to dashboard Cite

In this paper, we introduce discrete-time linear mean-field games subject to an infinite-horizon discounted-cost optimality criterion. The state space of a generic agent is a compact Borel space. At every time, each agent is randomly coupled with another agent via their dynamics and one-stage cost function, where this randomization is generated via the empirical distribution of their states (i.e., the mean-field term). Therefore, the transition probability and the one-stage cost function of each agent depend linearly on the mean-field term, which is the key distinction between classical mean-field games and linear mean-field games. Under mild assumptions, we show that the policy obtained from infinite population equilibrium is ε(N )-Nash when the number of agents N is sufficiently large, where ε(N ) is an explicit function of N . Then, using the linear programming formulation of MDPs and the linearity of the transition probability in mean-field term, we formulate the game in the infinite population limit as a generalized Nash equilibrium problem (GNEP) and establish an algorithm for computing equilibrium with a convergence guarantee. CONTENTS 1. Introduction. This paper introduces linear mean-field games, which are discrete-time stochastic dynamic games with one-stage cost and transition probability that are linear with respect to the empirical distribution of the states (i.e., mean-field term). Specifically, at each

show abstract

Section: Computing Equilibrium Of Gnepmentioning

confidence: 99%

Linear Mean-Field Games with Discounted Cost

Saldi¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Similar to Remark 2.2, when (2.7) holds, the stationary equilibrium concept in Definition 2.2 aligns with the conventional stationary equilibrium in time-consistent MFGs under the infinite horizon setting. This concept has been widely studied in the related literature, e.g., [20,1,3,29]. Now we characterize an MFG equilibrium in Definition 2.1 as a fixed point of the operator Φ ν below.…”

Section: Mean Field Games With Time-inconsistencymentioning

confidence: 99%

“…First, it has been shown that an equilibrium in the (time-consistent) MFG can serve as an approximate equilibrium in the (time-consistent) N -agent game when N is large enough; see, e.g. [13,26,3]. Secondly, efforts have also been made to show that equilibria in (time-consistent) N -agent games can approach an equilibrium in the (time-consistent) MFG as N tends to infinity; see, e.g., [18,22,23,17,6,12,5,4].…”

Section: Introductionmentioning

confidence: 99%

Finite State Mean Field Games with Wright–Fisher Common Noise as Limits ofN-Player Weighted Games

et al. 2022

View full text Add to dashboard Cite

Forcing finite state mean field games by a relevant form of common noise is a subtle issue, which has been addressed only recently. Among others, one possible way is to subject the simplex valued dynamics of an equilibrium by a so-called Wright–Fisher noise, very much in the spirit of stochastic models in population genetics. A key feature is that such a random forcing preserves the structure of the simplex, which is nothing but, in this setting, the probability space over the state space of the game. The purpose of this article is, hence, to elucidate the finite-player version and, accordingly, prove that N-player equilibria indeed converge toward the solution of such a kind of Wright–Fisher mean field game. Whereas part of the analysis is made easier by the fact that the corresponding master equation has already been proved to be uniquely solvable under the presence of the common noise, it becomes however more subtle than in the standard setting because the mean field interaction between the players now occurs through a weighted empirical measure. In other words, each player carries its own weight, which, hence, may differ from [Formula: see text] and which, most of all, evolves with the common noise.

show abstract

“…With the exception of [Angiuli et al, 2022b], which is the basis of the present paper, these methods focus on solving one of the two types of problems, MFG or MFC. On the one hand, to learn MFGs solutions, two classical families of methods are those relying on strict contraction and fixed point iterations (e.g., [Guo et al, 2019, Cui and Koeppl, 2021, Anahtarci et al, 2023 with tabular Qlearning or deep RL), and those relying on monotonicity and the structure of the game (e.g., , Perrin et al, 2020, Laurière et al, 2022 using fictitious play and tabular or deep RL). Two-timescale analysis to learn MFG solutions has been used in [Mguni et al, 2018, Subramanian andMahajan, 2019].…”

Section: Introductionmentioning

confidence: 99%

Unified reinforcement Q-learning for mean field game and control problems

Angiuli

Fouque

Laurière

2022

Math. Control Signals Syst.

View full text Add to dashboard Cite

We establish the convergence of the unified two-timescale Reinforcement Learning (RL) algorithm presented in [Angiuli et al., 2022b]. This algorithm provides solutions to Mean Field Game (MFG) or Mean Field Control (MFC) problems depending on the ratio of two learning rates, one for the value function and the other for the mean field term. We focus a setting with finite state and action spaces, discrete time and infinite horizon. The proof of convergence relies on a generalization of the two-timescale approach of [Borkar, 1997]. The accuracy of approximation to the true solutions depends on the smoothing of the policies. We then provide an numerical example illustrating the convergence. Last, we generalize our convergence result to a three-timescale RL algorithm introduced in [Angiuli et al., 2022a] to solve mixed Mean Field Control Games (MFCGs).

show abstract

Q-Learning in Regularized Mean-field Games

Cited by 18 publications

References 55 publications

Linear Mean-Field Games with Discounted Cost

Linear Mean-Field Games with Discounted Cost

Finite State Mean Field Games with Wright–Fisher Common Noise as Limits ofN-Player Weighted Games

Unified reinforcement Q-learning for mean field game and control problems

Contact Info

Product

Resources

About