SMIX(λ): Enhancing Centralized Value Functions for Cooperative Multiagent Reinforcement Learning

Yao, Xinghu; Chao, Wen Hung; Wang, Yuhui; Tan, Xiaoyang

doi:10.1109/tnnls.2021.3089493

Cited by 13 publications

(2 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1) Selection of Value Decomposition Network: After the advent of QMIX, many derivative algorithms [32], [34], [35] are based on the hyper-network to establish the connection between joint action-values and individual action values. However, in MCVD, the joint action-values is obtained by summing all individual action values, just like VDN does.…”

Section: Details Of Mcvdmentioning

confidence: 99%

Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learning

Li¹,

Zhang²,

Kong³

2022

Preprint

View full text Add to dashboard Cite

We explore value decomposition solutions for multiagent deep reinforcement learning in the popular paradigm of centralized training with decentralized execution(CTDE). As the recognized best solution to CTDE, Weighted QMIX is cutting-edge on StarCraft Multi-agent Challenge (SMAC), with a weighting scheme implemented on QMIX to place more emphasis on the optimal joint actions. However, the fixed weight requires manual tuning according to the application scenarios, which painfully prevents Weighted QMIX from being used in broader engineering applications. In this paper, we first demonstrate the flaw of Weighted QMIX using an ordinary One-Step Matrix Game (OMG), that no matter how the weight is chosen, Weighted QMIX struggles to deal with non-monotonic value decomposition problems with a large variance of reward distributions. Then we characterize the problem of value decomposition as an Underfitting One-edged Robust Regression problem and make the first attempt to give a solution to the value decomposition problem from the perspective of information-theoretical learning. We introduce the Maximum Correntropy Criterion (MCC) as a cost function to dynamically adapt the weight to eliminate the effects of minimum in reward distributions. We simplify the implementation and propose a new algorithm called MCVD. A preliminary experiment conducted on OMG shows that MCVD could deal with non-monotonic value decomposition problems with a large tolerance of kernel bandwidth selection. Further experiments are carried out on Cooperative-Navigation and multiple SMAC scenarios, where MCVD exhibits unprecedented ease of implementation, broad applicability, and stability.

show abstract

Section: Details Of Mcvdmentioning

confidence: 99%

Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learning

Li¹,

Zhang²,

Kong³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Hausknecht et al [17] proposed the DRQN algorithm, which replaces the fully connected layer of DQN with recurrent neural network and can better handle the missing information.In 2016, Wang et al [18] proposed the Dueling-DQN algorithm to optimize the internal of DQN framework and better results in performance evaluation. In the same year, DeepMind proposed the AlphaGo algorithm [19], which mainly utilizes Monte Carlo tree search and deep learning techniques, and then learns Go games through human prior knowledge, and soon AlphaGo defeated the Go master Lee Sedol, attracting widespread attention from the world.…”

Section: Introductionmentioning

confidence: 99%

Heritage and Innovation of Art Creation in the Context of Big Data and Public Health Events

2023

FAR

View full text Add to dashboard Cite

In the context of big data of public health events, the inheritance and innovation of fine art creation becomes a new opportunity and challenge. In this paper, based on the art image data collection, we establish a convolutional neural network model with migration learning and inverse algorithm for the recognition of art creation under the big data of public health events. The experimental results show that the visual image analysis algorithm can classify paintings with an accuracy of more than 99%, which has good painting recognition effect and classification ability. The migration learning model achieves a training accuracy of over 97% at 50 training rounds and also performs well on the test set with an accuracy of 96. 97%. Using the neural network model of art creation established in this paper for intelligent innovation of art images, with the increase of model settings, it can guarantee the recognition of art creation images between 60% and 80%, and with the increase of iterations, the effective linearity keeps stabilizing and the innovation rate keeps increasing, up to 79.0%. In this paper, the established methods are introduced into Lenet-s model, GoogleNet model and ResNet model for experiments, and it can be seen that the training sets1 of the three models have greater than or equal to 99.8%, 99.6% and 99.4%, respectively, and the training sets2 of the three models have 99.4%, 99.2% and 99.4%, respectively, which proves the neural network model of this paper validity.

show abstract

An Overestimation Reduction Method Based on the Multi-step Weighted Double Estimation Using Value-Decomposition Multi-agent Reinforcement Learning

Zhao,

Chang,

Guo

et al. 2024

Neural Process Lett

View full text Add to dashboard Cite

The joint action-value function (JAVF) plays a key role in the centralized training of multi-agent deep reinforcement learning (MADRL)-based algorithms using the value function decomposition (VFD) and in the generating process of a collaborative policy between agents. However, under the influence of multiple factors such as environmental noise, inadequate exploration and iterative updating mechanism, estimation bias is inevitably introduced, causing its overestimation problem, which in turn prevents agents from obtaining accurate reward signals during the learning process, and fails to correctly approximate the optimal policy. To address this problem, this paper first analyzes the causes of joint action-value function overestimation, gives the corresponding mathematical proofs and theoretical derivations, and obtains the lower bound of the overestimation error; then, a MADRL overestimation reduction method based on the multi-step weighted double estimation named λWD QMIX is proposed. Specifically, the λWD QMIX method effectively achieves more stable and accurate JAVF estimation results using the bias correction estimation mechanisms based on the weighted double estimation and multi-step updating based on eligibility trace backup, without additionally adding or changing any network structure. The results of a series of experiments on the StarCraft II micromanipulation benchmark show that the proposed λWD QMIX algorithm can effectively improve the final performance and learning efficiency of the baseline algorithm, and can be seamlessly integrated with the partially MADRL algorithms based on communication learning.

show abstract

SMIX(λ): Enhancing Centralized Value Functions for Cooperative Multiagent Reinforcement Learning

Cited by 13 publications

References 16 publications

Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learning

Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learning

Heritage and Innovation of Art Creation in the Context of Big Data and Public Health Events

An Overestimation Reduction Method Based on the Multi-step Weighted Double Estimation Using Value-Decomposition Multi-agent Reinforcement Learning

Contact Info

Product

Resources

About