Multi-Agent Constrained Policy Optimisation

Gu, Shangding; Kuba, Jakub Grudzien; Wen, Muning; Chen, Ruiqing; Wang, Ziyan; Tian, Zheng; Wang, Jun; Knoll, Alois; Yang, Yaodong

doi:10.48550/arxiv.2110.02793

Cited by 6 publications

(7 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They have been applied in the distribution network or microgrids to deal with the large-scale integrated renewable energy resources and voltage violations [23], [24]. However, algorithms always use a single value function or share the same state value function in multiagent RL which are basically trained in a centralized way [25]. They may suffer from heavy communication burdens, which is prone to communication failures.…”

Section: Introductionmentioning

confidence: 99%

Finite-Sample Analysis for Decentralized Batch Multiagent Reinforcement Learning With Networked Agents

Zhang

Yang

Liu

et al. 2021

IEEE Trans. Automat. Contr.

View full text Add to dashboard Cite

This paper proposes a multiagent based bi-level operation framework for the low-carbon demand management in distribution networks considering the carbon emission allowance on the demand side. In the upper level, the aggregate load agents optimize the control signals for various types of loads to maximize the profits; in the lower level, the distribution network operator makes optimal dispatching decisions to minimize the operational costs and calculates the distribution locational marginal price and carbon intensity. The distributed flexible load agent has only incomplete information of the distribution network and cooperates with other agents using networked communication. Finally, the problem is formulated into a networked multi-agent constrained Markov decision process, which is solved using a safe reinforcement learning algorithm called consensus multi-agent constrained policy optimization considering the carbon emission allowance for each agent. Case studies with the IEEE 33-bus and 123-bus distribution network systems demonstrate the effectiveness of the proposed approach, in terms of satisfying the carbon emission constraint on demand side, ensuring the safe operation of the distribution network and preserving privacy of both sides.

show abstract

Section: Introductionmentioning

confidence: 99%

Finite-Sample Analysis for Decentralized Batch Multiagent Reinforcement Learning With Networked Agents

Zhang

Yang

Liu

et al. 2021

IEEE Trans. Automat. Contr.

View full text Add to dashboard Cite

show abstract

“…When solving Dec-POMDPs or potential games , the framework of centralised training and decentralised execution (CTDE) is often employed Rashid et al, 2018;Wen et al, 2018;Hu et al, 2021) where a centralised critic is trained to gather all agents' local observations and assign credits. While CTDE methods rely on the individual-global-max assumption (Son et al, 2019), another thread of work is built on the so-called advantage decomposition lemma (Kuba et al, 2021b), which holds in general for any co-operative games; such a lemma leads to provably convergent multi-agent trust-region methods (Kuba et al, 2021a) and constrained policy optimisation methods (Gu et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks

Meng¹,

Wen²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Offline reinforcement learning leverages previously-collected offline datasets to learn optimal policies with no necessity to access the real environment. Such a paradigm is also desirable for multi-agent reinforcement learning (MARL) tasks, given the increased interactions among agents and with the enviroment. Yet, in MARL, the paradigm of offline pre-training with online fine-tuning has not been studied, nor datasets or benchmarks for offline MARL research are available. In this paper, we facilitate the research by providing large-scale datasets, and use it to examine the usage of the Decision Transformer in the context of MARL. We investigate the generatlisation of MARL offline pre-training in the following three aspects: 1) between single agents and multiple agents, 2) from offline pretraining to the online fine tuning, and 3) to that of multiple downstream tasks with few-shot and zero-shot capabilities. We start by introducing the first offline MARL dataset with diverse quality levels based on the StarCraftII environment, and then propose the novel architecture of multi-agent decision transformer (MADT) for effective offline learning. MADT leverages transformer's modelling ability of sequence modelling and integrates it seamlessly with both offline and online MARL tasks. A crucial benefit of MADT is that it learns generalisable policies that can transfer between different types of agents under different task scenarios. On StarCraft II offline dataset, MADT outperforms the state-of-the-art offline RL baselines. When applied to online tasks, the pre-trained MADT significantly improves sample efficiency, and enjoys strong performance both few-short and zero-shot cases. To our best knowledge, this is the first work that studies and demonstrates the effectiveness of offline pre-trained models in terms of sample efficiency and generalisability enhancements in MARL.

show abstract

“…The partial work of the study is shared on an open platform [21]. In this study, we provide a more comprehensive investigation and solution for safe MARL.…”

Section: Introductionmentioning

confidence: 99%