A Q-Learning-Based Approach for Simple and Multi-Agent Systems

Abstract

This study proposes different machine learning-based solutions to both single and multi-agent systems, took place on a 2-D simulation platform, namely, Robocode. This dynamic and programmable platform allows agents to interact with the environment and each other by employing a variety of battling strategies. Q-Learning is one of the leading and popular machine learning-based solutions to be applied to such a problem. However, especially for continued spaces, the control problem gets deeper. Essentially, one of the main drawbacks of reinforcement learning (RL) is to design an appropriate reward function that the function can be described by only employing few parameters for simple tasks, whereas estimating the goal of the reward function may be a challenging problem. Recent studies prove that neural network-based approaches can handle these challenges and achieve to learn control strategies from 2-D or 1-D data. Besides those problems of RL algorithms for single robots, once the number of robots increases and the systems need to behave as multi-agent systems, the overall design requirements become more complex. Accordingly, the proposed system is validated by considering different battle scenarios. The performance of the Q-Learning-based system and the supervised learning techniques are compared by employing different scenarios for this problem. Results reveal the superiority of the ANN-based approach over other methods.

A Novel Behavioral Strategy for RoboCode Platform Based on Deep Q‐Learning

“…Strategies are the most critical guidelines for how to act in a particular situation. Effective strategies to be implemented in the arena are of great importance for international tournaments that are organized annually for the RoboCode war simulator [13]. It should be noted that, while a robot can perform well in a one-on-one battle, other adaptive strategies may be required for close combat.…”

“…Q-learning is one of the leading off-policy RL algorithms, preferred in another recent study due to its efficiency and popularity. However, in this study, an artificial neural network is designed to approximate Q values instead of trying to keep them in a "Qtable," which is essentially not possible for such a continuous space problem [8]. e neural network has a very modest structure, involving only two layers.…”

“…Along with the first award estimated, the agent starts inserting the "q value" regarding the relation between the state and action into the Q-table. Subsequently, the agent starts to predict and reach the prize by maximizing the values of forward-looking moves at each iteration [8].…”

“…RoboCode is an open-source war simulator program developed by IBM using Java programming language in 2001 [12,13]. e purpose of this simulator is to program a war robot using the classes offered by the platform to the users in a two-dimensional environment and to measure the performance of the programmed agent by fighting these robots in the environment provided by this simulator which is called the arena.…”

“…In order for a robot to be a winner, it is not sufficient only to survive on all rounds because robots can earn more points than the winner due to some offensive actions. A list of the rules for the RoboCode platform can be seen in [8,9]. e flow diagram of the process cycle used by the RoboCode engine can also be seen in Figure 2.…”