Value Iteration Algorithm for Mean-field Games

Anahtarcı, Berkay; Karıksız, Can Deha; Saldi, Naci

doi:10.48550/arxiv.1909.01758

Cited by 7 publications

(11 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the above derivation, (1) and ( 2) are obtained from Assumption 8 and Eq. 84 (also refer Anahtarci, Kariksiz, and Saldi (2019)). Combining Eq.…”

Section: Proof Of Theoremmentioning

confidence: 95%

See 1 more Smart Citation

Decentralized Mean Field Games

Subramanian¹,

Taylor²,

Crowley³

et al. 2021

Preprint

View full text Add to dashboard Cite

Multiagent reinforcement learning algorithms have not been widely adopted in large scale environments with many agents as they often scale poorly with the number of agents. Using mean field theory to aggregate agents has been proposed as a solution to this problem. However, almost all previous methods in this area make a strong assumption of a centralized system where all the agents in the environment learn the same policy and are effectively indistinguishable from each other. In this paper, we relax this assumption about indistinguishable agents and propose a new mean field system known as Decentralized Mean Field Games, where each agent can be quite different from others. All agents learn independent policies in a decentralized fashion, based on their local observations. We define a theoretical solution concept for this system and provide a fixed point guarantee for a Q-learning based algorithm in this system. A practical consequence of our approach is that we can address a 'chicken-and-egg' problem in empirical mean field reinforcement learning algorithms. Further, we provide Q-learning and actor-critic algorithms that use the decentralized mean field learning approach and give stronger performances compared to common baselines in this area. In our setting, agents do not need to be clones of each other and learn in a fully decentralized fashion. Hence, for the first time, we show the application of mean field learning methods in fully competitive environments, large-scale continuous action space environments, and other environments with heterogeneous agents. Importantly, we also apply the mean field method in a ride-sharing problem using a real-world dataset. We propose a decentralized solution to this problem, which is more practical than existing centralized training methods.

show abstract

“…In the above derivation, (1) and ( 2) are obtained from Assumption 8 and Eq. 84 (also refer Anahtarci, Kariksiz, and Saldi (2019)). Combining Eq.…”

Section: Proof Of Theoremmentioning

confidence: 95%

“…In this and the subsequent theorems, we follow the theoretical results and proofs in the work by Anahtarci, Kariksiz, and Saldi (2019), and extend them to the decentralized setting.…”

Section: Proof Of Theoremmentioning

confidence: 96%

Decentralized Mean Field Games

Subramanian¹,

Taylor²,

Crowley³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…where q (y) t and q (z) t are given by (9). Let us denote y t = x u 1 ,u 2 t − xu 1 ,u 2 t and z t = xu 1 ,u 2 t for every t ≥ 0.…”

Section: Probabilistic Setupmentioning

confidence: 99%

“…As a cornerstone for applications, the development of numerical methods for these mean-field problems has also attracted a growing interest. Assuming full knowledge of the model, methods for which convergence guarantees have been established include finite difference schemes for partial differential equations [1,2], semi-Lagrangian schemes [22], augmented Lagrangian or primal-dual methods [5,17,18], value iteration algorithm [9], or neural network based stochastic methods [26,27]; see e.g. [6] for a recent overview.…”

Section: Introductionmentioning

confidence: 99%

Linear-Quadratic Zero-Sum Mean-Field Type Games: Optimality Conditions and Policy Optimization

Carmona¹,

Hamidouche²,

Laurière³

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper, zero-sum mean-field type games (ZSMFTG) with linear dynamics and quadratic cost are studied under infinite-horizon discounted utility function. ZSMFTG are a class of games in which two decision makers whose utilities sum to zero, compete to influence a large population of indistinguishable agents. In particular, the case in which the transition and utility functions depend on the state, the action of the controllers, and the mean of the state and the actions, is investigated. The optimality conditions of the game are analysed for both open-loop and closed-loop controls, and explicit expressions for the Nash equilibrium strategies are derived. Moreover, two policy optimization methods that rely on policy gradient are proposed for both model-based and sample-based frameworks. In the model-based case, the gradients are computed exactly using the model, whereas they are estimated using Monte-Carlo simulations in the sample-based case. Numerical experiments are conducted to show the convergence of the utility function as well as the two players' controls.

show abstract

“…We note that the aforementioned papers, except linear models, mostly identify the existence of mean-field equilibrium and no algorithm with convergence guarantee has been proposed to compute this mean-field equilibrium. In our recent work [24], this problem is explored for mean-field games with abstract state and action spaces under both discounted cost and average cost criteria. We have developed a value iteration algorithm and proved that this algorithm converges to the mean-field equilibrium.…”

Section: Introductionmentioning

confidence: 99%

Q-Learning in Regularized Mean-field Games

Anahtarcı¹,

Karıksız²,

Saldi³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we introduce a regularized mean-field game and study learning of this game under an infinite-horizon discounted reward function. The game is defined by adding a regularization function to the one-stage reward function in the classical mean-field game model. We establish a value iteration based learning algorithm to this regularized mean-field game using fitted Q-learning. This regularization term in general makes reinforcement learning algorithm more robust with improved exploration. Moreover, it enables us to establish error analysis of the learning algorithm without imposing restrictive convexity assumptions on the system components, which are needed in the absence of a regularization term.

show abstract

Value Iteration Algorithm for Mean-field Games

Cited by 7 publications

References 5 publications

Decentralized Mean Field Games

Decentralized Mean Field Games

Linear-Quadratic Zero-Sum Mean-Field Type Games: Optimality Conditions and Policy Optimization

Q-Learning in Regularized Mean-field Games

Contact Info

Product

Resources

About