Social Agents Playing a Periodical Policy

Abstract

Coordination is an important issue in multiagent systems. Within the stochastic game framework this problem translates to policy learning in a joint action space. This technique however suffers some important drawbacks like the assumption of the existence of a unique Nash equilibrium and synchronicity, the need for central control, the cost of communication, etc. Moreover in general sum games it is not always clear which policies should be learned. Playing pure Nash equilibria is often unfair to at least one of the players, while playing a mixed strategy doesn't give any guarantee for coordination and usually results in a sub-optimal payoff for all agents. In this work we show the usefulness of periodical policies, which arise as a side effect of the fairness conditions used by the agents. We are interested in games which assume competition between the players, but where the overall performance can only be as good as the performance of the poorest player. Players are social distributed reinforcement learners, who have to learn to equalize their payoff. Our approach is illustrated on synchronous one-step games as well as on asynchronous job scheduling games.

Exploring selfish reinforcement learning in repeated games with stochastic rewards

“…In a common interest game, ESRL is able to find one of the Pareto optimal solutions of the game. In a conflicting interest game, we show that ESRL agents learn optimal fair, possibly periodical policies [17,26]. Important to know is that ESRL agents are independent in the sense that they only use their own action choices and rewards to base their decisions on, that ESRL agents are flexible in learning different solution concepts and they can handle both stochastic, possible delayed rewards and asynchronous action selection.…”

“…We call a solution optimally fair when there is no other solution that is also fair for the agents but gives the agents more reward on average. Periodical policies were first introduced in [17].…”

Strategy and Fairness in Repeated Two-agent Interaction

“…We compare our results with a related periodical policy [5], and simulations show that agents using our adaptive strategy are able to achieve more optimal fairness results in the sense of obtaining higher utilitarian social welfare. Besides, the agents using our adaptive strategy can achieve fairness with less payoff cost compared with periodical policy when period length becomes smaller.…”

“…Nowé et al [5] [7] propose a periodical policy for achieving fair outcomes among multiple agents in a distributed way. This policy can be divided into two periods: reinforcement learning period and communication period.…”

“…In this section, the performance of the agents adopting our adaptive strategy is evaluated by comparing with that when using the periodical policy in [5]. The experimental setting follows the problem specification described in III-A and the payoff matrix for the two agents' conflicting interest game is instantiated as the one in Fig.1(a).…”