Q-Table compression for reinforcement learning

Abstract

Reinforcement learning (RL) algorithms are often used to compute agents capable of acting in environments without prior knowledge of the environment dynamics. However, these algorithms struggle to converge in environments with large branching factors and their large resulting state-spaces. In this work, we develop an approach to compress the number of entries in a Q-value table using a deep auto-encoder. We develop a set of techniques to mitigate the large branching factor problem. We present the application of such techniques in the scenario of a real-time strategy (RTS) game, where both state space and branching factor are a problem. We empirically evaluate an implementation of the technique to control agents in an RTS game scenario where classical RL fails and provide a number of possible avenues of further work on this problem.

A Low-Cost Q-Learning-Based Approach to Handle Continuous Space Problems for Decentralized Multi-Agent Robot Navigation in Cluttered Environments

“…Each robot observes the environment locally and takes the action without relying on a centralized controller for synchronization between robots thus not having any direct information about other robot's intensions. However, it is a good practice to hugely accelerate the learning process, alleviate branching factor problem [35] and gain more confidence on learned policies visiting most of the states in state space several times by sharing the observations of all robots only in learning phase without harming the decentralization concept in exploitation phase. We should bear in mind that sharing the observations is not a necessary task for our method.…”

“…However, our method does not rely on coordination between agents and despite QMIX, learning phase and running phase both are done in a decentralized fashion. There are some other researches that try to compress the state space and reduce dimensionality [35], [45]. These researches utilize a method named deep auto-encoders for compressing the state space utilizing neural networks, mainly with the aim of gaining generalization ability rather than memory usage reduction.…”

“…[49] provides a compression strategy for reducing the size of the experience by using an encoder and decoder to shrink the size of the streaming data in order to save bandwidth and power consumption. For sharing the q-table [35], [50] utilize a normalization step for merging the q-table of each robot after each learning episode which requires extra processing and also extra memory for holding number of times that each agent visited each state, e.g., if we name this table as visit-table in python language, selecting float type (16bits) for q-table and int (12 bits) for visit-table, it increases the memory usage by 75% multiplied by the number of agents, making it even more costly and even impractical for systems with large agent count. On the other hand, it has the advantage of learning in real-time during each episode.…”

Unsupervised Deep Learning: Taxonomy and algorithms

“…Artificial neural networks (ANNs) are a type of machine learning model made up of numerous nodes grouped in layers that compute an output depending on node activation mediated by weights in the connections between them. ANNs are capable of solving a variety of machine learning tasks, including classification, regression, and dimensionality reduction [44].…”

Special issue on adaptive and learning agents 2017

“…The fourth paper Q-table compression for reinforcement learning by Rosa Amado and Meneguzzi (2018) proposes a method to reduce the number of entries in a Q-value table by using a deep autoencoder. Multi-agent reinforcement learning where agents share experience updates is also applied to mitigate the large branching factors which are present when controlling teams of units in real-time strategy (RTS) games.…”