Structural Abstraction Experiments in Reinforcement Learning

Abstract

A challenge in applying reinforcement learning to large problems is how to manage the explosive increase in storage and time complexity. This is especially problematic in multi-agent systems, where the state space grows exponentially in the number of agents. Function approximation based on simple supervised learning is unlikely to scale to complex domains on its own, but structural abstraction that exploits system properties and problem representations shows more promise. In this paper, we investigate several classes of known abstractions: 1) symmetry, 2) decomposition into multiple agents, 3) hierarchical decomposition, and 4) sequential execution. We compare memory requirements, learning time, and solution quality empirically in two problem variations. Our results indicate that the most effective solutions come from combinations of structural abstractions, and encourage development of methods for automatic discovery in novel problem formulations.

Intelligent Multiagent Coordination Based On Neuro-Fuzzy Models With Hierarchical Reinforcement Learning

“…Thus, the main purpose of coordination is to avoid or minimize these problems while optimizing the resources and time to perform complex tasks. Such tasks can be completed though a single agent does not have all of the resources that are necessary to complete the task or meet several sub-goals to achieve the overall goal [1] [2].…”

“…Through parallel computing, multiple agents can work together to better exploit the decentralized structure of a given task and accelerate its completion [1] [2]. Additionally, agents can exchange experiences by communicating [3], observe and learn from the most skilled agents [4], and serve as teachers for other agents [5].…”

Cooperative Multi-Agent Reinforcement Learning for Multi-Component Robotic Systems: guidelines for future research

“…Fuzzy theory has also been applied to obtain abstractions of state sets and generalize over them [4,5,21]. Some authors have also empirically studied di erent manually set state abstraction operations, such as [25] which studied symmetry and multi-agency homomorphic mappings. Homomorphisms may allow to reduce the size of MDPs, but they do not guarantee that the reduced problem is relevant to solve the original one.…”

Memory Based on Abstraction for Dynamic Fitness Functions

“…Thus, abstraction means a threshold for and compression of information, see e.g. [8] which proposes similar ideas for reinforcement learning. When an environment change is detected, the stored abstraction information is extracted to generate new individuals into the population.…”