Value iteration algorithm for mean-field games

Anahtarcı, Berkay; Karıksız, Can Deha; Saldi, Naci

doi:10.1016/j.sysconle.2020.104744

Cited by 11 publications

(10 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recall that sup x∈X g(x) ≤ diam(X) and sup x∈X g 2 (x) ≤ diam(X) 2 . Therefore, we can bound the last term as follows…”

Section: Recall the Optimality Equation For Mdpmentioning

confidence: 99%

“…The previous studies reviewed only establish the existence and uniqueness of the meanfield equilibrium, but do not provide an algorithm with a guarantee of convergence to compute it, with the exception of the model with linear state dynamics and two other papers [41,2]. This work investigates this problem for linear mean-field games and proposes an algorithm that formulates the game as a GNEP, proving the convergence of the algorithm to the stationary mean-field equilibrium.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Linear Mean-Field Games with Discounted Cost

Saldi¹

2023

Preprint

View full text Add to dashboard Cite

In this paper, we introduce discrete-time linear mean-field games subject to an infinite-horizon discounted-cost optimality criterion. The state space of a generic agent is a compact Borel space. At every time, each agent is randomly coupled with another agent via their dynamics and one-stage cost function, where this randomization is generated via the empirical distribution of their states (i.e., the mean-field term). Therefore, the transition probability and the one-stage cost function of each agent depend linearly on the mean-field term, which is the key distinction between classical mean-field games and linear mean-field games. Under mild assumptions, we show that the policy obtained from infinite population equilibrium is ε(N )-Nash when the number of agents N is sufficiently large, where ε(N ) is an explicit function of N . Then, using the linear programming formulation of MDPs and the linearity of the transition probability in mean-field term, we formulate the game in the infinite population limit as a generalized Nash equilibrium problem (GNEP) and establish an algorithm for computing equilibrium with a convergence guarantee. CONTENTS 1. Introduction. This paper introduces linear mean-field games, which are discrete-time stochastic dynamic games with one-stage cost and transition probability that are linear with respect to the empirical distribution of the states (i.e., mean-field term). Specifically, at each

show abstract

“…Recall that sup x∈X g(x) ≤ diam(X) and sup x∈X g 2 (x) ≤ diam(X) 2 . Therefore, we can bound the last term as follows…”

Section: Recall the Optimality Equation For Mdpmentioning

confidence: 99%

mentioning

confidence: 99%

Linear Mean-Field Games with Discounted Cost

Saldi¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For linear-quadratic mean field games, Li and Zhang [30] employed the fixed point method to study mean field games with time-averaged stochastic cost functionals and introduced the notion of decentralized asymptotic Nash equilibrium in the sense of probability; Bardi [2] discussed the case with time-averaged deterministic cost functionals by the direct method developed in [26][27][28]; Huang and Zhou [23] studied the asymptotic solvability of linear-quadratic mean field games by establishing a scale reset method; Bensoussan et al [3] considered linear-quadratic mean field games based on the stochastic maximum principle. For nonlinear mean field games, Anahtarci et al [1] developed value iteration algorithms to investigate mean field games with discounted cost functionals and time-averaged cost functionals respectively, then they proved that the sequence of strategies designed is a decentralized asymptotic Nash equilibrium.…”

Section: Introductionmentioning

confidence: 99%

“…We use the subscript L and subscript F as the label of major and minor players. For a vector or matrix X, X ⊤ denotes the transposition of X, ∥X∥ denotes the 2norm of vector or Frobenius norm of matrix, and Tr(X) denotes the trace of X; I n denotes the n×n-dimensional identity matrix, R n denotes the set of n-dimensional real column vectors, and R n×m denotes the set of n × mdimensional real matrices; C 1,2…”

Section: Introductionmentioning

confidence: 99%

Risk-sensitive mean field games with major and minor players

Chen

Xin

2023

ESAIM: COCV

View full text Add to dashboard Cite

We investigate a class of mean field games containing a large number of major and minor players. Each player minimizes a quadratic-tracking type risk-sensitive cost functional, where the reference signal is a function of the state average term of the major and minor players. To reduce the complexity for solving the problem, we design a sequence of decentralized strategies by the Nash certainty equivalence principle. Firstly, for the optimal control problems with quadratic type risksensitive cost functionals, we propose a new verification theorem. Secondly, we apply the two-layer state aggregation method to construct the fixed-point equations for the estimations of the state average terms and give the conditions for the existence and uniqueness of the fixed points. Then, we design a sequence of decentralized strategies by the estimations of the state average terms based on local information. It is shown that the estimations of the state average terms are consistent with the true values for the closed-loop systems, and the sequence of strategies designed is a decentralized asymptotic Nash equilibrium. Finally, the effectiveness of the theoretical analysis is demonstrated by a numerical example.

show abstract

“…As a cornerstone for applications, the development of numerical methods for these mean-field problems has also attracted a growing interest. Assuming full knowledge of the model, methods for which convergence guarantees have been established include finite difference schemes for partial differential equations [1,2], semi-Lagrangian schemes [22], augmented Lagrangian or primal-dual methods [5,17,18], value iteration algorithm [9], or neural network based stochastic methods [26,27]; see e.g. [4] for a recent overview.…”

mentioning

confidence: 99%

Linear-quadratic zero-sum mean-field type games: Optimality conditions and policy optimization

Carmona

Hamidouche

Laurière

et al. 2021

JDG

View full text Add to dashboard Cite

<p style='text-indent:20px;'>In this paper, zero-sum mean-field type games (ZSMFTG) with linear dynamics and quadratic cost are studied under infinite-horizon discounted utility function. ZSMFTG are a class of games in which two decision makers whose utilities sum to zero, compete to influence a large population of indistinguishable agents. In particular, the case in which the transition and utility functions depend on the state, the action of the controllers, and the mean of the state and the actions, is investigated. The optimality conditions of the game are analysed for both open-loop and closed-loop controls, and explicit expressions for the Nash equilibrium strategies are derived. Moreover, two policy optimization methods that rely on policy gradient are proposed for both model-based and sample-based frameworks. In the model-based case, the gradients are computed exactly using the model, whereas they are estimated using Monte-Carlo simulations in the sample-based case. Numerical experiments are conducted to show the convergence of the utility function as well as the two players' controls.</p>

show abstract

Value iteration algorithm for mean-field games

Cited by 11 publications

References 23 publications

Linear Mean-Field Games with Discounted Cost

Linear Mean-Field Games with Discounted Cost

Risk-sensitive mean field games with major and minor players

Linear-quadratic zero-sum mean-field type games: Optimality conditions and policy optimization

Contact Info

Product

Resources

About