A Novel Multi-Agent Parallel-Critic Network Architecture for Cooperative-Competitive Reinforcement Learning

Abstract

multi-agent deep reinforcement learning (MDRL) is an emerging research hotspot and application direction in the field of machine learning and artificial intelligence. MDRL covers many algorithms, rules and frameworks, it is currently researched in swarm system, energy allocation optimization, stocking analysis, sequential social dilemma, and with extremely bright future. In this paper, a parallel-critic method based on classic MDRL algorithm MADDPG is proposed to alleviate the training instability problem in cooperative-competitive multi-agent environment. Furthermore, a policy smoothing technique is introduced to our proposed method to decrease the variance of learning policies. The suggested method is evaluated in three different scenarios of authoritative multi-agent particle environment (MPE). Multiple statistical data of experimental results show that our method significantly improves the training stability and performance compared to vanilla MADDPG.INDEX TERMS multi-agent system, deep reinforcement learning, parallel-critic architecture, training stability.

A Fast Optimal Coordination Method for Multiagent in Complex Environment

“…Te distance between the agent and the corresponding target point is inversely proportional to the obtained reward. As shown in Table 2, a comparison is made between the proposed approach and existing advanced learning methods [19,30,31,33] across six aspects: initial reward, maximum reward, stable reward, growth reward, training time (episode), and standardized variance.…”

“…Te second is to change the evaluation strategy of the critic network, for higher response speed together with less calculation errors [15,28,29]. With similar focus, the methods proposed in [19,30,31] take into account the characteristics of these two types of methods and presented excellent performances: the growth reward, training speed, and stability of these methods are signifcantly better than other methods. In reference [19], a strategy smoothing technique is introduced into the MADDPG method to reduce the variance of learning strategies so as to alleviate the training instability of cooperative and competitive multiagent and signifcantly improve the stability and performance of training.…”

“…With similar focus, the methods proposed in [19,30,31] take into account the characteristics of these two types of methods and presented excellent performances: the growth reward, training speed, and stability of these methods are signifcantly better than other methods. In reference [19], a strategy smoothing technique is introduced into the MADDPG method to reduce the variance of learning strategies so as to alleviate the training instability of cooperative and competitive multiagent and signifcantly improve the stability and performance of training. Te authors in reference [30] introduced a new intrinsic motivation mechanism (GICM), which encourages agents to pursue novel and surprising states to explore the current scene more comprehensively.…”

“…De Souza et al [17] used deep reinforcement learning to pursue an omnidirectional goal and realized pursuit and evasion actions between multiagents. Te authors in reference [18,19] used the MADDPG method to solve the problem of agent cooperation in MPE environment. Tampuu et al [20] brought the deep quantum learning framework to a multiagent environment and studied the interaction and cooperative competition between two learning agents in the famous video game Pong.…”

Decentralized multi-agent reinforcement learning based on best-response policies

“…As pointed out by Ackermann et al (2019) , the overestimation bias is also present in MARL. Some initial works have proposed to bridge concepts from the single-agent domain ( van Hasselt, 2010 ) to MARL ( Sun et al, 2020 ). Thus, SAC has been adjusted to the multi-agent domain by Wei et al (2018) , for which further extensions have been outlined, e.g., Zhang et al (2020) proposed a Lyapunov-based penalty term to the policy update to stabilize the policy gradient.…”