Exploring selfish reinforcement learning in repeated games with stochastic rewards

Abstract

In this paper we introduce a new multi-agent reinforcement learning algorithm, called exploring selfish reinforcement learning (ESRL). ESRL allows agents to reach optimal solutions in repeated non-zero sum games with stochastic rewards, by using coordinated exploration. First, two ESRL algorithms for respectively common interest and conflicting interest games are presented. Both ESRL algorithms are based on the same idea, i.e. an agent explores by temporarily excluding some of the local actions from its private action space, to give the team of agents the opportunity to look for better solutions in a reduced joint action space. In a latter stage these two algorithms are transformed into one generic algorithm which does not assume that the type of the game is known in advance. ESRL is able to find the Pareto optimal solution in common interest games without communication. In conflicting interest games ESRL only needs limited communication to learn a fair periodical policy, resulting in a good overall policy. Important to know is that ESRL agents are independent in the sense that they only use their own action choices and rewards to base their decisions on, that ESRL agents are flexible in learning different solution concepts and they can handle both stochastic, possible delayed rewards and asynchronous action selection. A real-life experiment, i.e. adaptive load-balancing of parallel applications is added.K. Verbeeck (B) Computational Modeling Lab (COMO), Vrije Universiteit Brussel, Brussels, Belgium

Speeding up learning automata based multi agent systems using the concepts of stigmergy and entropy

“…In the past few years, a significant part of the research has focused on comprehending and solving single-stage multi-agent problems, modeled as a normal form game (Verbeeck, Nowé, Parent, & Tuyls, 2007) and multi-stage games modeled as Markov games (Verbeeck, Nowé, Peeters, & Tuyls, 2005). Wheeler et al have shown that a set of decentralized learning automata is able to control a finite Markov Chain with unknown transition probabilities and rewards (Wheeler & Narendra, 1986).…”

Multiagent Learning in Large Anonymous Games

“…Unfortunately, these algorithms involve searching the whole strategy space, so their convergence time is exponential. Another algorithm that uses stages to provide a stable learning environment is the ESRL algorithm for coordinated exploration (Verbeeck, Nowé, Parent, & Tuyls, 2007). Marden, Arslan, and Shamma (2007b) and Marden, Young, Arslan, and Shamma (2009) use an algorithm with experimentation and best replies but without explicit stages that converges for weakly acyclic games, where best-reply dynamics converge when agents move one at a time, rather than moving all at once, as we assume here.…”

Long-term fairness with bounded worst-case losses

“…In previous literature [17,18], distributed algorithms for discovering such sequences found suboptimal ones. We will show optimal solutions, albeit with non-distributed algorithms.…”