Q-Learning in Regularized Mean-field Games

Anahtarcı, Berkay; Karıksız, Can Deha; Saldi, Naci

doi:10.48550/arxiv.2003.12151

Cited by 9 publications

(18 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With a general dynamics T , the propagation of the distribution of µ N t could be computationally expensive. On the other hand, under the identical policy π used by all agents, the trajectory of the mean field is deterministic [21]. Furthermore, µ t follows a simple propagation rule:…”

Section: Mean Field Approximationmentioning

confidence: 99%

See 1 more Smart Citation

Shaping Large Population Agent Behaviors Through Entropy-Regularized Mean-Field Games

Guan

Zhou

Pakniyat

et al. 2021

Preprint

View full text Add to dashboard Cite

Mean-field games (MFG) were introduced to efficiently analyze approximate Nash equilibria in large population settings. In this work, we consider entropy-regularized meanfield games with a finite state-action space in a discrete time setting. We show that entropy regularization provides the necessary regularity conditions, that are lacking in the standard finite mean field games. Such regularity conditions enable us to design fixed-point iteration algorithms to find the unique meanfield equilibrium (MFE). Furthermore, the reference policy used in the regularization provides an extra means, through which one can control the behavior of the population. We first formulate the problem as a stochastic game with a large population of N homogeneous agents. We establish conditions for the existence of a Nash equilibrium in the limiting case as N tends to infinity, and we demonstrate that the Nash equilibrium for the infinite population case is also an -Nash equilibrium for the N -agent regularized game, where the sub-optimality is of order O 1/ √ N . Finally, we verify the theoretical guarantees through a resource allocation example and demonstrate the efficacy of using a reference policy to control the behavior of a large population of agents.

show abstract

Section: Mean Field Approximationmentioning

confidence: 99%

“…More recent works directly introduced a relative entropy term to the reward structure to provide regularity conditions. The existence of stationary entropy-regularized MFE was examined in [21]. The authors in [19] studied transient MFGs with finite horizon.…”

Section: Introductionmentioning

confidence: 99%

Shaping Large Population Agent Behaviors Through Entropy-Regularized Mean-Field Games

Guan

Zhou

Pakniyat

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To approximate stationary MFG solutions, (Guo et al, 2019) uses fixed point iterations on the distribution combined with Q-learning to learn the best response at each iteration. (Anahtarci et al, 2020) combines this kind of learning scheme together with an entropic regularization. Convergence of an actor-critic method for linear-quadratic MFG has been studied in (Fu et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

Reinforcement Learning for Mean Field Games, with Applications to Economics

Angiuli¹,

Fouque²,

Laurière³

2021

Preprint

View full text Add to dashboard Cite

Mean field games (MFG) and mean field control problems (MFC) are frameworks to study Nash equilibria or social optima in games with a continuum of agents. These problems can be used to approximate competitive or cooperative games with a large finite number of agents and have found a broad range of applications, in particular in economics. In recent years, the question of learning in MFG and MFC has garnered interest, both as a way to compute solutions and as a way to model how large populations of learners converge to an equilibrium. Of particular interest is the setting where the agents do not know the model, which leads to the development of reinforcement learning (RL) methods. After reviewing the literature on this topic, we present a two timescale approach with RL for MFG and MFC, which relies on a unified Q-learning algorithm. The main novelty of this method is to simultaneously update an action-value function and a distribution but with different rates, in a model-free fashion. Depending on the ratio of the two learning rates, the algorithm learns either the MFG or the MFC solution. To illustrate this method, we apply it to a mean field problem of accumulated consumption in finite horizon with HARA utility function, and to a trader's optimal liquidation problem.

show abstract

“…However, there is virtually no theoretical study on the role of entropy regularization in multiagent RL (MARL), with the exception of [2]. Indeed, most existing studies are empirical, demonstrating convergence improvement and variance reduction when entropy regularization is added.…”

Section: Introductionmentioning

confidence: 99%

“…For instance, [12] showed via empirical analysis that policy features can be learned directly from pure observations of other agents and that the non-stationarity of the environment can be reduced with addition of the cross-entropy; [11] applied the cross-entropy regularization to demonstrate the convergence of fictitious play in a discrete-time model with a finite number of agents while [22] used the cross-entropy loss to train the prediction of other agents' actions via observations of their behavior. The only theoretical work so far can be found in [2] with an infinite horizon setting in which a regularized Q-learning algorithm for stationary discrete-time mean field game was proposed along with its convergence analysis. Still, the problem remains open in the finite time horizon cases, which arises often in many applications.…”

Section: Introductionmentioning

confidence: 99%

Entropy Regularization for Mean Field Games with Learning

Guo

Zariphopoulou

2020

Preprint

View full text Add to dashboard Cite

Entropy regularization has been extensively adopted to improve the efficiency, the stability, and the convergence of algorithms in reinforcement learning. This paper analyzes both quantitatively and qualitatively the impact of entropy regularization for Mean Field Game (MFG) with learning in a finite time horizon. Our study provides a theoretical justification that entropy regularization yields time-dependent policies and, furthermore, helps stabilizing and accelerating convergence to the game equilibrium. In addition, this study leads to a policy-gradient algorithm for exploration in MFG. Under this algorithm, agents are able to learn the optimal exploration scheduling, with stable and fast convergence to the game equilibrium.

show abstract

Q-Learning in Regularized Mean-field Games

Cited by 9 publications

References 17 publications

Shaping Large Population Agent Behaviors Through Entropy-Regularized Mean-Field Games

Shaping Large Population Agent Behaviors Through Entropy-Regularized Mean-Field Games

Reinforcement Learning for Mean Field Games, with Applications to Economics

Entropy Regularization for Mean Field Games with Learning

Contact Info

Product

Resources

About