Negotiating team formation using deep reinforcement learning

Bachrach, Yoram; Everett, Richard; Hughes, Edward; Lazaridou, Angeliki; Leibo, Joel Z.; Lanctot, Marc; Johanson, Michael; Czarnecki, Wojciech Marian; Graepel, Thore

doi:10.1016/j.artint.2020.103356

Cited by 25 publications

(16 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This graph is typically assumed to be a design parameter, as in this work. A separate but related body of works seek to estimate the communications architecture when agents' behavior is fixed [16,1,3], or begin with locality-based correlation models to derive dependencies on agents' local utilities [49,38].…”

Section: Discussionmentioning

confidence: 99%

MARL with General Utilities via Decentralized Shadow Reward Actor-Critic

Zhang,

Bedi,

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

We posit a new mechanism for cooperation in multi-agent reinforcement learning (MARL) based upon any nonlinear function of the team's long-term state-action occupancy measure, i.e., a general utility. This subsumes the cumulative return but also allows one to incorporate risk-sensitivity, exploration, and priors. We derive the Decentralized Shadow Reward Actor-Critic (DSAC) in which agents alternate between policy evaluation (critic), weighted averaging with neighbors (information mixing), and local gradient updates for their policy parameters (actor). DSAC augments the classic critic step by requiring agents to (i) estimate their local occupancy measure in order to (ii) estimate the derivative of the local utility with respect to their occupancy measure, i.e., the "shadow reward". DSAC converges to -stationarity in O(1/ 2.5 ) (Theorem 4.14) or faster O(1/ 2 ) (Corollary 4.16) steps with high probability, depending on the amount of communications. We further establish the non-existence of spurious stationary points for this problem, that is, DSAC finds the globally optimal policy (Corollary 4.15). Experiments demonstrate the merits of goals beyond the cumulative return in cooperative MARL.Preprint. Under review.

show abstract

Section: Discussionmentioning

confidence: 99%

MARL with General Utilities via Decentralized Shadow Reward Actor-Critic

Zhang,

Bedi,

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The Banzhaf index of a i can be viewed as the expected increase in performance under uncertainty about the participation of other players in the team 1 The Shapley value has also been used to examine power in team formation (Aziz et al, 2009;Mash et al, 2017;Bachrach et al, 2020), combinatorial tasks (Ueda et al, 2011;Banarse et al, 2019), pricing and auctions (Bachrach, 2010;Kamboj et al, 2011;Blocq et al, 2014) or political settings (Bilbao et al, 2002;Bachrach et al, 2011;Filmus et al, 2019), or feature importance for model explainability (Lundberg and ).…”

Section: Preliminaries: Cooperative Game Theorymentioning

confidence: 99%

Game-theoretic Vocabulary Selection via the Shapley Value and Banzhaf Index

Patel¹,

Garnelo²,

Gemp³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

The input vocabulary and their learned representations are crucial to the performance of neural NLP models. Using the full vocabulary results in less explainable and more memory intensive models, with the embedding layer often constituting the majority of model parameters. It is thus common to use a smaller vocabulary to lower memory requirements and construct more interpertable models.We propose a vocabulary selection method that views words as members of a team trying to maximize the model's performance. We apply power indices from cooperative game theory, including the Shapley value and Banzhaf index, that measure the relative importance of individual team members in accomplishing a joint task. We approximately compute these indices to identify the most influential words.Our empirical evaluation examines multiple NLP tasks, including sentence and document classification, question answering and textual entailment. We compare to baselines that select words based on frequency, TF-IDF and regression coefficients under L1 regularization, and show that this game-theoretic vocabulary selection outperforms all baselines on a range of different tasks and datasets. * Work done during an internship at DeepMind.

show abstract

“…Multi-agent reinforcement learning and related techniques have been applied across a very large spectrum of environments, ranging from board games (Tesauro, 1994;Silver et al, 2016;Moravcik et al, 2017;Brown & Sandholm, 2018;Jaderberg et al, 2019;Anthony et al, 2020;Gray et al, 2020) and computer games (Mnih et al, 2015;Vinyals et al, 2019;Berner et al, 2019), simulations of economic settings and social dilemmas (Leibo et al, 2017;Lerer & Peysakhovich, 2017;Hughes et al, 2020;Zheng et al, 2020), negotiation (Lewis et al, 2017;Bachrach et al, 2020), and simulated or real-world multi-robot systems (Yang & Gu, 2004;Kober et al, 2013;Stone et al, 2005;Banarse et al, 2019;Liu et al, 2019;Baker et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

Evaluating Strategic Structures in Multi-Agent Inverse Reinforcement Learning

Fu¹,

Tacchetti²,

Pérolat³

et al. 2021

jair

Self Cite

View full text Add to dashboard Cite

A core question in multi-agent systems is understanding the motivations for an agent's actions based on their behavior. Inverse reinforcement learning provides a framework for extracting utility functions from observed agent behavior, casting the problem as finding domain parameters which induce such a behavior from rational decision makers. We show how to efficiently and scalably extend inverse reinforcement learning to multi-agent settings, by reducing the multi-agent problem to N single-agent problems while still satisfying rationality conditions such as strong rationality. However, we observe that rewards learned naively tend to lack insightful structure, which causes them to produce undesirable behavior when optimized in games with different players from those encountered during training. We further investigate conditions under which rewards or utility functions can be precisely identified, on problem domains such as normal-form and Markov games, as well as auctions, where we show we can learn reward functions that properly generalize to new settings.

show abstract

Negotiating team formation using deep reinforcement learning

Cited by 25 publications

References 53 publications

MARL with General Utilities via Decentralized Shadow Reward Actor-Critic

MARL with General Utilities via Decentralized Shadow Reward Actor-Critic

Game-theoretic Vocabulary Selection via the Shapley Value and Banzhaf Index

Evaluating Strategic Structures in Multi-Agent Inverse Reinforcement Learning

Contact Info

Product

Resources

About