Reinforcement Learning Exploration Algorithms for Energy Harvesting Communications Systems

Abstract

Prolonging the lifetime, and maximizing the throughput are important factors in designing an efficient communications system, especially for energy harvesting-based systems. In this work, the problem of maximizing the throughput of point-to-point energy harvesting communications system, while prolonging its lifetime is investigated. This work considers more real communications system, where this system does not have a priori knowledge about the environment. This system consists of a transmitter and receiver. The transmitter is equipped with an infinite buffer to store data, and energy harvesting capability to harvest renewable energy and store it in a finite battery. The problem of finding an efficient power allocation policy is formulated as a reinforcement learning problem. Two different exploration algorithms are used, which are the convergence-based and the epsilon-greedy algorithms. The first algorithm uses the action-value function convergence error and the exploration time threshold to balance between exploration and exploitation. On the other hand, the second algorithm tries to achieve balancing through the exploration probability (i.e. epsilon). Simulation results show that the convergence-based algorithm outperforms the epsilon-greedy algorithm. Then, the effects of the parameters of each algorithm are investigated.

Sensing, Computing, and Communications for Energy Harvesting IoTs: A Survey

“…Ayatollahi et al [139] considered a MIMO system where the transmitter can change the number of antennas during transmission and employed Q-learning to learn the optimal transmission policy. Masadeh et al [140] utilized SARSA algorithm to investigate the exploration and exploitation balancing problem and demonstrated that convergence-based algorithm outperforms the epsilon-greedy algorithm.…”

Distributed Power Control for Large Energy Harvesting Networks: A Multi-Agent Deep Reinforcement Learning Approach

“…Next, at the (m + 1) th iteration, each node attempts to learn the Nash maximizer, F * m+1 . A discrete-time MFG is said to have FPP if and only if the procedure described by (14), (15) and (16) converges.…”

“…This is because, the distributed DNN approach does not use the estimate of the distribution π for learning the policy, i.e., in the distributed DNN approach π is used only for sampling the states of the other nodes. Also, from (16), regardless of the update frequency, over the time, iterative estimates of the distribution π converge, and the states of the other nodes are sampled from the correct distribution. In contrast, for the MF-MARL, estimates of π are critically used for learning, therefore the estimation inaccuracies jeopardize the learning procedure, and may adversely affect the sumthroughput.…”

Deep Reinforcement Learning-Based Access Control for Buffer-Aided Relaying Systems With Energy Harvesting

“…One of the promising approaches used is reinforcement learning (RL), which is known as algorithms enabling to optimize system performance in unknown environments [15], [16]. In [13], an EH point-to-point communications system is investigated. The EH and channel gain processes are modeled as Markov processes and Q-learning was used to learn a transmission power allocation policy that maximizes the amount of data arriving at the destination.…”