On the Complexity of Computing Markov Perfect Equilibrium in General-Sum Stochastic Games

Deng, Xin; Li, Yuhao; Mguni, David; Wang, Jun; Yang, Yaodong

doi:10.48550/arxiv.2109.01795

Cited by 8 publications

(7 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has been typically applied to selfdriving (Shalev-Shwartz et al, 2016;, order dispatching , modeling population dynamics , and gaming AIs (Peng et al, 2017;Zhou et al, 2021). However, the scheme of learning policy from experience requires the algorithms with high computational complexity (Deng et al, 2021) and sample efficiency due to the limited computing resources and high cost resulting from the data collection (Haarnoja et al, 2018;Munos et al, 2016;Espeholt et al, 2019;. Furthermore, even in domains where the online environment is feasible, we might still prefer to utilise previously-collected data instead; for example, if the domain's complex requires large datasets for effective generalisation.…”

Section: Inrtoductionmentioning

confidence: 99%

Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks

Meng¹,

Wen²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Offline reinforcement learning leverages previously-collected offline datasets to learn optimal policies with no necessity to access the real environment. Such a paradigm is also desirable for multi-agent reinforcement learning (MARL) tasks, given the increased interactions among agents and with the enviroment. Yet, in MARL, the paradigm of offline pre-training with online fine-tuning has not been studied, nor datasets or benchmarks for offline MARL research are available. In this paper, we facilitate the research by providing large-scale datasets, and use it to examine the usage of the Decision Transformer in the context of MARL. We investigate the generatlisation of MARL offline pre-training in the following three aspects: 1) between single agents and multiple agents, 2) from offline pretraining to the online fine tuning, and 3) to that of multiple downstream tasks with few-shot and zero-shot capabilities. We start by introducing the first offline MARL dataset with diverse quality levels based on the StarCraftII environment, and then propose the novel architecture of multi-agent decision transformer (MADT) for effective offline learning. MADT leverages transformer's modelling ability of sequence modelling and integrates it seamlessly with both offline and online MARL tasks. A crucial benefit of MADT is that it learns generalisable policies that can transfer between different types of agents under different task scenarios. On StarCraft II offline dataset, MADT outperforms the state-of-the-art offline RL baselines. When applied to online tasks, the pre-trained MADT significantly improves sample efficiency, and enjoys strong performance both few-short and zero-shot cases. To our best knowledge, this is the first work that studies and demonstrates the effectiveness of offline pre-trained models in terms of sample efficiency and generalisability enhancements in MARL.

show abstract

Section: Inrtoductionmentioning

confidence: 99%

Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks

Meng¹,

Wen²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…A well-known solution concept that describes the equilibrium of NF games is Nash Equilibrium (NE) [11,30]. Let p i (𝑠) where…”

Section: Preliminaries and Notationsmentioning

confidence: 99%

Measuring the Non-Transitivity in Chess

Sanjaya¹,

Wang²,

Yang³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

quantifications-Nash Clustering and counting the number of Rock-Paper-Scissor cycles-on over one billion match data from Lichess and FICS. Our findings positively indicate that the strategy space occupied by real-world Chess strategies demonstrates a spinning top geometry, and more importantly, there exists a strong connection between the degree of non-transitivity and the progression of a Chess player's rating. In particular, high degrees of non-transitivity tend to prevent human players from making progress on their Elo rating, whereas progressions are easier to make at the level of ratings where the degree of non-transitivity is lower. Additionally, we also investigate the implication of the degree of non-transitivity for population-based training methods. By considering fixed-memory Fictitious Play as a proxy, we reach the conclusion that maintaining large-size and diverse populations of strategies is imperative to training effective AI agents in solving Chess types of games.

show abstract

“…Stochastic games (SGs) (Shapley 1953;Deng et al 2021) offer a multi-player game framework where agents jointly decide the loss and the state transition. Compared to OMDPs, the main difference is that SGs allow each player to have representation of states, actions and rewards, thus players can learn the representations over time and find the NE of the stochastic games (Wei, Hong, and Lu 2017;Tian et al 2020).…”

Section: Related Workmentioning

confidence: 99%

Online Markov Decision Processes with Non-oblivious Strategic Adversary

Dinh,

Mguni,

Tran-Thanh

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We study a novel setting in Online Markov Decision Processes (OMDPs) where the loss function is chosen by a nonoblivious strategic adversary who follows a no-external regret algorithm. In this setting, we first demonstrate that MDP-Expert, an existing algorithm that works well with oblivious adversaries can still apply and achieve a policy regret bound of O ( log(where is the size of adversary's pure strategy set and | | denotes the size of agent's action space. Considering real-world games where the support size of a NE is small, we further propose a new algorithm: MDP-Online Oracle Expert (MDP-OOE), that achieves a policy regret bound of O ( log( ) + 2 log( )) where depends only on the support size of the NE. MDP-OOE leverages the key benefit of Double Oracle in game theory and thus can solve games with prohibitively large action space. Finally, to better understand the learning dynamics of no-regret methods, under the same setting of no-external regret adversary in OMDPs, we introduce an algorithm that achieves last-round convergence result to a NE. To our best knowledge, this is first work leading to the last iteration result in OMDPs.

show abstract

On the Complexity of Computing Markov Perfect Equilibrium in General-Sum Stochastic Games

Cited by 8 publications

References 33 publications

Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks

Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks

Measuring the Non-Transitivity in Chess

Online Markov Decision Processes with Non-oblivious Strategic Adversary

Contact Info

Product

Resources

About