Agent-Level Maximum Entropy Inverse Reinforcement Learning for Mean Field Games

Chen, Yang; Zhang, Libo; Liu, Jiamou; Witbrock, Michael

doi:10.48550/arxiv.2104.14654

Cited by 4 publications

(10 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the Investment-Graphon problem -an adaptation of a problem studied by Chen et al (2021), where it was in turn adapted from Weintraub et al ( 2010) -we consider many firms maximizing profits, where profits are proportional to product quality and decrease with total neighborhood product quality, i.e. the graph models overlap in e.g.…”

Section: Methodsmentioning

confidence: 99%

Learning Graphon Mean Field Games and Approximate Nash Equilibria

Cui¹,

Koeppl²

2021

Preprint

View full text Add to dashboard Cite

Recent advances at the intersection of dense large graph limits and mean field games have begun to enable the scalable analysis of a broad class of dynamical sequential games with large numbers of agents. So far, results have been largely limited to graphon mean field systems with continuous-time diffusive or jump dynamics, typically without control and with little focus on computational methods. We propose a novel discrete-time formulation for graphon mean field games as the limit of non-linear dense graph Markov games with weak interaction. On the theoretical side, we give extensive and rigorous existence and approximation properties of the graphon mean field solution in sufficiently large systems. On the practical side, we provide general learning schemes for graphon mean field equilibria by either introducing agent equivalence classes or reformulating the graphon mean field system as a classical mean field system. By repeatedly finding a regularized optimal control solution and its generated mean field, we successfully obtain plausible approximate Nash equilibria in otherwise infeasible large dense graph games with many agents. Empirically, we are able to demonstrate on a number of examples that the finite-agent behavior comes increasingly close to the mean field behavior for our computed equilibria as the graph or system size grows, verifying our theory. More generally, we successfully apply policy gradient reinforcement learning in conjunction with sequential Monte Carlo methods.

show abstract

Section: Methodsmentioning

confidence: 99%

Learning Graphon Mean Field Games and Approximate Nash Equilibria

Cui¹,

Koeppl²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…ERMFNE serves as a generalization of MaxEnt IRL to MFGs. The researchers show that this method is effectively able to recover ground truth rewards for MFGs [19].…”

Section: Entropy Regularized Mean Field Nash Equilibriummentioning

confidence: 99%

“…Computational intensity and algorithmic run-time for computing traditional NE solution concepts grows exponentially with expansion of joint stateaction spaces when population size increases [48,15,23]. A MFNE solution concept simplifies this computation for a population model through considering its asymptotic limit, assuming that agents in the population are homogeneous, and that the population approaches infinity [19]. Mean-field approximations leverage the empirical distribution representing aggregated population behaviors, reducing interactions to a dual-view interplay of a single agent and the whole population.…”

Section: Entropy Regularized Mean Field Nash Equilibriummentioning

confidence: 99%

See 1 more Smart Citation

Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Concepts

Bergerson

2021

Preprint

View full text Add to dashboard Cite

Multi-agent inverse reinforcement learning (MIRL) can be used to learn reward functions from agents in social environments. To model realistic social dynamics and outcomes, MIRL methods must account for suboptimal human reasoning and behavior. Traditional formalisms of game theory provide computationally tractable behavioral models, but assume agents have unrealistic cognitive capabilities. The objective of this research is to identify and compare mechanisms used in MIRL methods to a) handle the presence of noise, biases and heuristics in agent decision making and b) model realistic equilibrium solution concepts based on suboptimal behavior. MIRL research was systematically reviewed to identify solution attempts for these challenges. The methods and results of these studies were analyzed and compared based on factors including performance accuracy, efficiency, and descriptive quality. We found that the primary methods for handling noise, biases and heuristics in MIRL were extensions of Maximum Entropy (MaxEnt) IRL to multi-agent settings. We also found that traditional Nash equilibrium (NE) solution concepts were ill-suited to model human behavior, and more success could be found with generalization of the NE. These solutions include the correlated equilibrium, logistic stochastic best response equilibrium and entropy regularized mean field NE. Methods which use recursive reasoning or updating also perform well, including the feedback NE and archive multi-agent adversarial IRL. Success in modeling specific biases and heuristics in single-agent IRL and promising results using a Theory of Mind approach in MIRL imply that modeling specific biases and heuristics in MIRL may be useful. Flexibility and unbiased inference in the identified alternative solution concepts suggest that a solution concept which has both recursive and generalized characteristics may perform well at modeling realistic social interactions.

show abstract

“…In this paper, we come up a new result which assures that even with non-uniform interactions, MFC is a good choice for approximating MARL if the reward of each agent is an affine function of the mean-field distributions 'seen' by that agent. We note that the behaviour of agents in multitude of social and economic networks can be modeled via affine rewards (refer the examples given in (Chen et al, 2021)), and thus for many cases of practical interest, MFC can approximate MARL with non-uniform interactions.…”

Section: Introductionmentioning

confidence: 99%

Can Mean Field Control (MFC) Approximate Cooperative Multi Agent Reinforcement Learning (MARL) with Non-Uniform Interaction?

Mondal¹,

Aggarwal²,

Ukkusuri³

2022

Preprint

View full text Add to dashboard Cite

Mean-Field Control (MFC) is a powerful tool to solve Multi-Agent Reinforcement Learning (MARL) problems. Recent studies have shown that MFC can well-approximate MARL when the population size is large and the agents are exchangeable. Unfortunately, the presumption of exchangeability implies that all agents uniformly interact with one another which is not true in many practical scenarios. In this article, we relax the assumption of exchangeability and model the interaction between agents via an arbitrary doubly stochastic matrix. As a result, in our framework, the mean-field 'seen' by different agents are different. We prove that, if the reward of each agent is an affine function of the mean-field seen by that agent, then one can approximate such a non-uniform MARL problem via its associated MFC problem within an error of e = O( 1where N is the population size and |X |, |U| are the sizes of state and action spaces respectively. Finally, we develop a Natural Policy Gradient (NPG) algorithm that can provide a solution to the non-uniform MARL with an error O(max{e, ǫ}) and a sample complexity of O(ǫ −3 ) for any ǫ > 0.

show abstract

Agent-Level Maximum Entropy Inverse Reinforcement Learning for Mean Field Games

Cited by 4 publications

References 17 publications

Learning Graphon Mean Field Games and Approximate Nash Equilibria

Learning Graphon Mean Field Games and Approximate Nash Equilibria

Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Concepts

Can Mean Field Control (MFC) Approximate Cooperative Multi Agent Reinforcement Learning (MARL) with Non-Uniform Interaction?

Contact Info

Product

Resources

About