Learning robotic soccer controllers with the Q-Batch update-rule

Abstract

Robotic soccer provides a rich environment for the development of Reinforcement Learning controllers. The competitive environment imposes strong requirements on performance of the developed controllers. RL offers a valuable alternative for the development of efficient controllers while avoiding the hassle of parameter tuning a hand coded policy. This paper presents the application of a recently proposed Batch RL updaterule to learn robotic soccer controllers in the context of the RoboCup Middle Size League. The Q-Batch update-rule exploits the episodic structure of the data collection phase of Batch RL to efficiently evaluate and improve the learned policy. Three different learning tasks, with increasing difficulty, were developed and applied on a simulated environment and later on the physical robot. The performance of the learned controllers is mostly compared to hand-tuned controllers while some comparisons with other RL methods were performed. Results show that the proposed approach is able to learn the tasks in a reduced amount of time, even outperforming existing hand-coded solutions.

Architecting centralized coordination of soccer robots based on principle solution

“…In this kind of leagues the decision about role assignation is distributed among the players, and the selection of behaviors is decided on by the robot itself [8]. In the case of distributed robot soccer teams, however, some research has been focused on the learning behaviors of the agents using techniques such as reinforcement learning to control behaviors [9], learning behaviors and learning to dribble [10]. Other strategies are modeled and implemented using finite state machines [11], in some cases in combination with Petri Net Plans [12].…”

Arquitecturas Centralizadas de Coordinación. Extrapolación del Fútbol de Robots al Control de Modo de Operaciones de Micro-Redes

“…En[100] proponen el aprendizaje de comportamientos como lo son evasión de obstáculos y seguimiento de trayectorias en un robot tipo humanoide utilizado en la RoboCup 3D Simulation League, combinando tanto Redes Neuronales como Algoritmos Genéticos. En otro ejemplo utilizan un algoritmo Q-Learning combinado con una técnica basada en Batch Reinforcement Learning para el aprendizaje de comportamientos en un robot de la RoboCup Middle Size League, enfocando estos algoritmos en el control a bajo nivel del robot[101]. En[102], utilizan un algoritmo de Batch Reinforcement Learning para el aprendizaje de tareas como rotar alrededor de un punto, disparar el balón o recibir un pase en un…”