A selection-mutation model for q-learning in multi-agent systems

Tuyls, Karl; Verbeeck, Katja; Lenaerts, Tom

doi:10.1145/860575.860687

Cited by 70 publications

(41 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section we briefly repeat the main results of [21]. In this paper however, we will extend the results of that previous work.…”

Section: The Q-learning Dynamicssupporting

confidence: 57%

“…More precisely, we will apply these results to stochastic Dispersion Games. The experiments of [21] were conducted in one-stage games. In this paper we will extend this approach to multi-state onestage games, i.e.…”

Section: The Q-learning Dynamicsmentioning

confidence: 99%

“…Doing this allows us to plot the direction field for each state. This is a general approach to reduce stochastic payoffs to ED we can plot as in [21].…”

Section: Applying the Q-learning Dynamics To Stochastic Gamesmentioning

confidence: 99%

See 2 more Smart Citations

Analyzing Multi-agent Reinforcement Learning Using Evolutionary Dynamics

Hoen

Tuyls

2004

Machine Learning: ECML 2004

Self Cite

View full text Add to dashboard Cite

Abstract. In this paper, we show how the dynamics of Q-learning can be visualized and analyzed from a perspective of Evolutionary Dynamics (ED). More specifically, we show how ED can be used as a model for Qlearning in stochastic games. Analysis of the evolutionary stable strategies and attractors of the derived ED from the Reinforcement Learning (RL) application then predict the desired parameters for RL in MultiAgent Systems (MASs) to achieve Nash equilibriums with high utility. Secondly, we show how the derived fine tuning of parameter settings from the ED can support application of the COllective INtelligence (COIN) framework. COIN is a proved engineering approach for learning of cooperative tasks in MASs. We show that the derived link between ED and RL predicts performance of the COIN framework and visualizes the incentives provided in COIN toward cooperative behavior.

show abstract

“…In this section we briefly repeat the main results of [21]. In this paper however, we will extend the results of that previous work.…”

Section: The Q-learning Dynamicssupporting

confidence: 57%

Section: The Q-learning Dynamicsmentioning

confidence: 99%

See 1 more Smart Citation

Analyzing Multi-agent Reinforcement Learning Using Evolutionary Dynamics

Hoen

Tuyls

2004

Machine Learning: ECML 2004

Self Cite

View full text Add to dashboard Cite

show abstract

“…Many such tools are particularly suited to only certain learning methods, and only a few offer a common framework for multiple learning techniques. Evolutionary game theory is the main tool in this second category, and it was successfully used to study the properties of cooperative coevolution [78,296], to visualize basins of attraction to Nash equilibria for cooperative coevolution [191], and to study trajectories of concurrent Q-learning processes [271,263]. Another tool for modeling and predicting the dynamics of concurrent multi-agent learners was recently proposed in Vidal and Durfee [276,277].…”

Section: The Dynamics Of Learningmentioning

confidence: 99%

Cooperative Multi-Agent Learning: The State of the Art

Panait

Luke

2005

Auton Agent Multi-Agent Syst

987

559

View full text Add to dashboard Cite

Cooperative multi-agent systems are ones in which several agents attempt, through their interaction, to jointly solve tasks or to maximize utility. Due to the interactions among the agents, multi-agent problem complexity can rise rapidly with the number of agents or their behavioral sophistication. The challenge this presents to the task of programming solutions to multi-agent systems problems has spawned increasing interest in machine learning techniques to automate the search and optimization process.We provide a broad survey of the cooperative multi-agent learning literature. Previous surveys of this area have largely focused on issues common to specific subareas (for example, reinforcement learning or robotics). In this survey we attempt to draw from multi-agent learning work in a spectrum of areas, including reinforcement learning, evolutionary computation, game theory, complex systems, agent modeling, and robotics.We find that this broad view leads to a division of the work into two categories, each with its own special issues: applying a single learner to discover joint solutions to multi-agent problems (team learning), or using multiple simultaneous learners, often one per agent (concurrent learning). Additionally, we discuss direct and indirect communication in connection with learning, plus open issues in task decomposition, scalability, and adaptive dynamics. We conclude with a presentation of multi-agent learning problem domains, and a list of multi-agent learning resources.

show abstract

“…A selection-mutation model of Boltzmann Q-learning (Eqs. 1 and 2) has been proposed by Tuyls et al [36]. The dynamical system can again be decomposed into terms for exploitation (selection following the replicator dynamics) and exploration (mutation through randomization based on the Boltzmann mechanism):…”

Section: Evolutionary Models Of Multi-agent Learningmentioning

confidence: 99%

Learning in Networked Interactions: A Replicator Dynamics Approach

Bloembergen

Caliskanelli

Tuyls

2015

Artificial Life and Intelligent Agents

Self Cite

View full text Add to dashboard Cite

Abstract. Many real-world scenarios can be modelled as multi-agent systems, where multiple autonomous decision makers interact in a single environment. The complex and dynamic nature of such interactions prevents hand-crafting solutions for all possible scenarios, hence learning is crucial. Studying the dynamics of multi-agent learning is imperative in selecting and tuning the right learning algorithm for the task at hand. So far, analysis of these dynamics has been mainly limited to normal form games, or unstructured populations. However, many multi-agent systems are highly structured, complex networks, with agents only interacting locally. Here, we study the dynamics of such networked interactions, using the well-known replicator dynamics of evolutionary game theory as a model for learning. Different learning algorithms are modelled by altering the replicator equations slightly. In particular, we investigate lenience as an enabler for cooperation. Moreover, we show how well-connected, stubborn agents can influence the learning outcome. Finally, we investigate the impact of structural network properties on the learning outcome, as well as the influence of mutation driven by exploration.

show abstract

A selection-mutation model for q-learning in multi-agent systems

Cited by 70 publications

References 9 publications

Analyzing Multi-agent Reinforcement Learning Using Evolutionary Dynamics

Analyzing Multi-agent Reinforcement Learning Using Evolutionary Dynamics

Cooperative Multi-Agent Learning: The State of the Art

Learning in Networked Interactions: A Replicator Dynamics Approach

Contact Info

Product

Resources

About