A New Multi-Agent Reinforcement Learning Method Based on Evolving Dynamic Correlation Matrix

Abstract

Multi-agent reinforcement learning approaches can be roughly classified into two categories. One is the agent-based approach which can be implemented in real distributed systems, though most approaches of this type cannot provide meaningful theoretical verifications. The other can be seen as the more formalized approach, which can provide theoretical results. However, most of current algorithms usually require unrealistic global communication, which makes them impractical for real distributed systems. In this article, we propose a dynamic correlation matrix based multi-agent reinforcement learning approach where the meta-parameters are evolved using an evolutionary algorithm. We believe that our approach is able to fill the gap between the two kinds of traditional multi-agent reinforcement learning approaches by providing both agent-level implementation and system-level convergence verification. The basic idea of this approach is that agents learn not only from local environmental feedback, i.e., their own experiences and rewards, but also from other agents' experiences. In this way, the agents' learning speed can be increased significantly. The performance of the proposed algorithm is demonstrated on a number of application scenarios, including blackjack games, urban traffic control systems and multi-robot foraging.

Reinforcement Learning in Urban Network Traffic Signal Control: A Systematic Literature Review

“…illustrated the usefulness of modeling human decisions by Cumulative Prospect Theory (CPT) paradigm in RL and suggested that CPT-based criteria is useful in a NTSC application. (Gan et al, 2019) proposed a dynamic correlation matrix based MARL approach where the meta-parameters are evolved using an evolutionary algorithm in a distributed manner. This was done to provide meaningful theoretical verification by using both agent-level implementation and system-level convergence verification.…”

Aplicação de Aprendizado por Reforço no Blackjack: Estudo de Caso de Estimação de Parâmetros

“…Com o intuito de comprovar a eficácia dos parâmetros estimados α = 0,30 e γ = 0,15, foram reproduzidas mais 10épocas de 100 episódios cada. O resultado desses jogos foi comparado com desempenhos oriundos de outras combinações de parâmetros, estabelecidas na literatura, tanto noâmbito do Blackjack (Pérez-Uribe and Sanchez, 1998;Kakvi, 2009;Gan et al, 2019), quanto em outros tipos de aplicações (Celiberto Jr et al, 2012;Ottoni et al, 2018). A Tabela 4 mostra os valores dos parâmetros usados para comparação, assim como a aplicação para a qual foram estimados.…”

“…Certamente, a capacidade de um agente aprender com recompensas e penalidades agindo em um ambienteé um fator que contribui para viabilizar a utilização em games. No entanto, a literatura ainda carece de pesquisas que avaliem os efeitos da definição dos parâmetros do AR em aplicações em jogos, mais especificamente para domínios de jogos de cartas, como Blackjack (Pérez- Uribe and Sanchez, 1998;Kakvi, 2009;Gan et al, 2019).…”

An Enhanced Model-Free Reinforcement Learning Algorithm to Solve Nash Equilibrium for Multi-Agent Cooperative Game Systems

“…In recent years, reinforcement learning (RL) is one of the areas that have attracted the most research and development interest. RL maps the relationship between the learning state and behavior of agents, involving how for agents to choose their behavior in a dynamic environment to optimize the sum of cumulative rewards [11], [12], [13]. Many algorithmic ideas of RL can be applied to the consistency research of multi-agents [7], [14], [15].…”