Applications of Reinforcement Learning (RL) are increasingly relevant in the context of future networks. However, although there is a great availability of environments for other research areas such as computer vision, there is a lack of good ones for telecommunication purposes. This paper proposes a Beam-Selection MiniGrid, based on Minimalist Gridworld environment.

Reinforcement Learning (RL) is a promising alternative to traditional methods of user scheduling and beamselection (SBS). Most of the current works on this topic adopt deep RL, in which neural networks allow to adopt state and action spaces with dimensions larger than the ones supported by tabular RL. However, while deep RL uses approximations that prevent them from getting policies that can be guaranteed to be optimal, tabular RL allows methods that find the optimal policy. The lack of optimal solutions complicates the proper interpretation and assessment of results in deep RL applied to SBS. This paper discusses how optimal policies can be found in the context of SBS and the associated issues. It also provides environments based on finite Markov decision processes that promote reproducible results and support a smooth transition from simple to more advanced RL problems. The presented experiments provide a benchmark of state-of-art deep and tabular RL algorithms, including scenarios for which the optimal solution is known. The results indicate that there is still room for improvement concerning deep RL algorithms, which do not reach the optimal solution in the adopted scenarios. This methodology not only provides insight into the performance of RL methods but helps compare new algorithms by first looking at contrived problems and later expanding the number of states and actions.