TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations

Abstract

Deep reinforcement learning (DRL) has achieved super-human performance on complex video games (e.g., StarCraft II and Dota II). However, current DRL systems still suffer from challenges of multi-agent coordination, sparse rewards, stochastic environments, etc. In seeking to address these challenges, we employ a football video game, e.g., Google Research Football (GRF), as our testbed and develop an end-to-end learning-based AI system (denoted as TiKick 23 ) to complete this challenging task. In this work, we first generated a large replay dataset from the self-playing of single-agent experts, which are obtained from league training. We then developed a distributed learning system and new offline algorithms to learn a powerful multi-agent AI from the fixed single-agent dataset. To the best of our knowledge, Tikick is the first learning-based AI system that can take over the multi-agent Google Research Football full game, while previous work could either control a single agent or experiment on toy academic scenarios. Extensive experiments further show that our pre-trained model can accelerate the training process of the modern multi-agent algorithm and our method achieves state-of-theart performances on various academic scenarios. * Equal contribution 2 Codes can be found at https://github.com/TARTRL/TiKick.3 Videos available at https://sites.google.com/view/tikick.Preprint. Under review.

T109 Evaluation of the GEM® Premier™ 5000 with intelligent quality management 2 (IQM®2) at Beijing Tsinghua Changgung Hospital, Tsinghua University (Beijing, China)

“…The model of Tikick is released and our evaluation result of Tikick is consistent with the paper [15]. The results are shown in Figure 5, where FXP achieves the largest goal difference against all reference policies.…”

“…Finally, with large-scale training, we use FXP to solve the challenging 11-vs-11 multi-agent full game in Google Research Football (GRF) [17]. We compare our methods with SOTA models including the hardest built-in AI, PSRO w. BD&RD [20] agent, and Tikick agent [15]. FXP achieves over 94% win rate against available models with a significant goal difference.…”

“…Google Research Football (GRF) [17] is another mixed cooperativecompetitive game where very few works have shown strong performances in the 11-vs-11 full game. Tikick [15] trains the first learning-based agent that can take over the full game by imitation learning. [20] uses a diversity-aware and online variant of PSRO and defeats the hardest built-in bot.…”

“…As an alternative approach, we evaluate FXP and other models by playing against a set of unseen reference policies and compare their performance. We use GRF's built-in models with different levels as the reference policies and compare FXP with SOTA models including the hardest built-in AI, a PSRO-based agent, PSRO w. BD&RD [20], an imitation learning agent Tikick [15]. Note that since the PSRO w. BD&RD [20] never release their code or model.…”

“…Finally, we scale up FXP to tackle the challenging 11-vs-11 multi-agent full game in the Google Research Football (GRF) [17] environment. We compare the FXP agent with the hardest built-in AI, an imitation-learning agent [15], and a PSRO-based agent [20] and achieve higher goal differences than all baselines against reference policies of different levels. We also let FXP play against available models including built-in hard AI and Tikick, and achieve over 94% win rates with goal differences over 2.7.…”