Learning Complementary Multiagent Behaviors: A Case Study

Abstract

Abstract. As machine learning is applied to increasingly complex tasks, it is likely that the diverse challenges encountered can only be addressed by combining the strengths of different learning algorithms. We examine this aspect of learning through a case study grounded in the robot soccer context. The task we consider is Keepaway, a popular benchmark for multiagent reinforcement learning from the simulation soccer domain. Whereas previous successful results in Keepaway have limited learning to an isolated, infrequent decision that amounts to a turn-taking behavior (passing), we expand the agents' learning capability to include a much more ubiquitous action (moving without the ball, or getting open), such that at any given time, multiple agents are executing learned behaviors simultaneously. We introduce a policy search method for learning "GetOpen" to complement the temporal difference learning approach employed for learning "Pass". Empirical results indicate that the learned GetOpen policy matches the best hand-coded policy for this task, and outperforms the best policy found when Pass is learned. We demonstrate that Pass and GetOpen can be learned simultaneously to realize tightly-coupled soccer team behavior.

An Empirical Study of Potential-Based Reward Shaping and Advice in Complex, Multi-Agent Systems

“…This has been previously studied with a combined temporal difference and policy search solution [15]. Alternatively, learning just the behaviour of keepers without the ball would also be a multi-agent learning problem, provided games of 3v2 or more players, as at all times at least two keepers would be off-theball.…”

“…Our baseline learning keeper learns the GetOpen behaviour of keepers not in possession of the ball, originally introduced by Kalyanakrishnan and Stone [15]. Keepers in the original work [15] the ball and one for while not.…”

“…Keepers in the original work [15] the ball and one for while not. Ours will only learn the policy of keepers off-the-ball.…”

“…Ours will only learn the policy of keepers off-the-ball. This limitation was necessary because, as suggested in the original work [15], our early experimental testing showed that learning both the behaviours of keepers with the ball and without is infeasible with temporal difference solutions to both.…”

“…To overcome the difficulty of learning both policies by temporal difference, the existing work [15] updates a single off-the-ball policy with experiences from all agents. Each agent would then make action decisions by a policy search method using the globally available network.…”