RoboCup Simulation Leagues: Enabling Replicable and Robust Investigation of Complex Robotic Systems

Abstract

Physically-realistic simulated environments are powerful platforms for enabling measurable, replicable and statistically-robust investigation of complex robotic systems. Such environments are epitomised by the RoboCup simulation leagues, which have been successfully utilised to conduct massivelyparallel experiments in topics including: optimisation of bipedal locomotion, self-localisation from noisy perception data and planning complex multi-agent strategies without direct agent-to-agent communication. Many of these systems are later transferred to physical robots, making the simulation leagues invaluable well-beyond the scope of simulated soccer matches. In this study, we provide an overview of the RoboCup simulation leagues and describe their properties as they pertain to replicable and robust robotics research. To demonstrate their utility directly, we leverage the ability to run parallelised experiments to evaluate different competition formats (e.g. round robin) for the RoboCup 2D simulation league. Our results demonstrate that a previously-proposed hybrid format minimises fluctuations from 'true' (statistically-significant) team performance rankings within the time constraints of the RoboCup world finals. Our experimental analysis would be impossible with physical robots alone, and we encourage other researchers to explore the potential for enriching their experimental pipelines with simulated components, both to minimise experimental costs and enable others to replicate and expand upon their results in a hardware-independent manner.

Fractals2019: Combinatorial Optimisation with Dynamic Constraint Annealing

“…Not surprisingly, this assignment which was optimised against the world champion of 2018, is assortative: it prefers to place the fastest players in defence (defenders 5, 4, 3, 2, with the wing defenders 5 and 4 assigned the best types), followed by midfielders (7,6,8), and leaving the weakest types for forwards (10,11,9). To re-iterate, this is carried out at the transition step from Gliders2d-v2.2 to v2.3, which is still a relatively weak team overall.…”

“…The RoboCup Soccer 2D Simulation League provides a rich dynamic environment, facilitated by the RoboCup Soccer Simulator (RCSS), aimed to test advances in decentralised collective behaviours of autonomous agents. The challenges include concurrent adversarial actions, computational nondeterminism, noise and latency in asynchronous perception and actuation, and limited processing time [3,5,7,29,37,38,42,43,46]. The League progress has been supported by several important base code releases, covering both low-level skills and standardised world models of simulated agents [1,22,45,47].…”

Disruptive Innovations in RoboCup 2D Soccer Simulation League: From Cyberoos’98 to Gliders2016

“…On the one hand, the ability to run a massive number of simulated games on supercomputing clusters producing replicable results will only strengthen in time [4], and so may lend some hope in meeting this challenge positively. On the other hand, the enormous size and dimensionality of the search-space would defy any unstructured exploration strategy.…”

RoboCup 2D Soccer Simulation League: Evaluation Challenges

“…Using the ranking estimation methodology established by [18,19], we conducted an 8-team round-robin tournament for top 8 teams from RoboCup-2016. The estimation process used the released binaries of top RoboCup-2016 teams 7 , where all 28 pairs of teams play approximately 4000 games against one another.…”