Learning to Share in Multi-Agent Reinforcement Learning

Abstract

In this paper, we study the problem of networked multi-agent reinforcement learning (MARL), where a number of agents are deployed as a partially connected network and each interacts only with nearby agents. Networked MARL requires all agents make decision in a decentralized manner to optimize a global objective with restricted communication between neighbors over the network. Inspired by the fact that sharing plays a key role in human's learning of cooperation, we propose LToS, a hierarchically decentralized MARL framework that enables agents to learn to dynamically share reward with neighbors so as to encourage agents to cooperate on the global objective. For each agent, the high-level policy learns how to share reward with neighbors to decompose the global objective, while the low-level policy learns to optimize local objective induced by the high-level policies in the neighborhood. The two policies form a bi-level optimization and learn alternately. We empirically demonstrate that LToS outperforms existing methods in both social dilemma and networked MARL scenario.Preprint.

Learning Roles with Emergent Social Value Orientations

“…We selected the aforementioned role learning algorithm, ROMA (Wang et al, 2020), and sanction-based algorithm, LIO (Yang et al, 2020), as the baselines because our RESVO draws on the core ideas of them in the SVO-based role emergence and the role-based policy optimization 6 respectively. In addition, we selected several aforementioned SVO-based algorithms, including LToS (Yi et al, 2021) and D3C (Gemp et al, 2022) as baselines. Below we will briefly introduce the core idea of each algorithm again.…”

“…A reward-sharing agent i learns a individual orientation function, w i ηi : O i × A −i → R N , parameterized by η i , that maps its own observation o i and all other agents' actions a −i to a vector of reward-sharing ratios for all N agents. Unlike the existing methods based on SVO (Peysakhovich and Lerer, 2018b;Baker, 2020;Gemp et al, 2022;Yi et al, 2021) or saction (Koster et al, 2020;Lupu and Precup, 2020;Yang et al, 2020;Vinitsky et al, 2021;Dong et al, 2021) mechanism, the orientation function in RESVO (1) allows agents to reward itself 3 , and (2) the sum of all sharing ratios does not need to be equal to 1. This is one of the reasons why reward sharing, a mechanism used by existing work, can encourage the division of labor and solve ISD.…”

“…We compare our method against various baselines. For ROMA (Wang et al, 2020), LIO (Yang et al, 2020), LToS (Yi et al, 2021), and D3C (Gemp et al, 2022), we use the codes provided by the authors and the hyper-parameters that have been fine-tuned on the Sequential Social Dilemma Games (SSDG) (Vinitsky et al, 2019). The ablations LToS+r and D3C+r use the exact implementation and hyperparameter settings as LToS and D3C, respectively, but with the only difference that the role emergence mechanism is introduced, which is the same as in our method.…”

“…Inspired by the interdependence theory, many previous works have introduced the SVO into MARL to solve the ISD (Peysakhovich and Lerer, 2018b;Hughes et al, 2018a;Zhang et al, 2019;Wang et al, 2019;Baker, 2020;Gemp et al, 2022;Yi et al, 2021;Ivanov et al, 2021;Schmid et al, 2021). Peysakhovich and Lerer (2018b) introduces the SVO into MARL for the first time and proposes the concept of prosocial, that is, cooperative orientation agents.…”

“…Going a step further, D3C (Gemp et al, 2022) optimizes the linear combination weights by using the ratio of the worst equilibrium to the optimal solution (Price of Anarchy, PoA) that measures the quality of the equilibrium points. Concurrent work LToS (Yi et al, 2021) models the optimization problem of linearly transforming weights as a bi-level problem and uses an end-to-end approach to train weights and policies jointly. Considering the noise or privacy issues that instantaneous rewards for SVO modeling may introduce, some recent works shape the agents' reward in other ways.…”