Distributed Power Control for Large Energy Harvesting Networks: A Multi-Agent Deep Reinforcement Learning Approach

Abstract

In this paper, we develop a multi-agent reinforcement learning (MARL) framework to obtain online power control policies for a large energy harvesting (EH) multiple access channel, when only causal information about the EH process and wireless channel is available. In the proposed framework, we model the online power control problem as a discrete-time mean-field game (MFG), and analytically show that the MFG has a unique stationary solution. Next, we leverage the fictitious play property of the mean-field games, and the deep reinforcement learning technique to learn the stationary solution of the game, in a completely distributed fashion. We analytically show that the proposed procedure converges to the unique stationary solution of the MFG. This, in turn, ensures that the optimal policies can be learned in a completely distributed fashion. In order to benchmark the performance of the distributed policies, we also develop a deep neural network (DNN) based centralized as well as distributed online power control schemes. Our simulation results show the efficacy of the proposed power control policies. In particular, the DNN based centralized power control policies provide a very good performance for large EH networks for which the design of optimal policies is intractable using the conventional methods such as Markov decision processes. Further, performance of both the distributed policies is close to the throughput achieved by the centralized policies.

Deep-Learning-Based Wireless Resource Allocation With Application to Vehicular Networks

“…That said, such a combination of environment nonstationarity and partial observability makes learning extremely difficult, which is made even worse if we scale the number of users large as in the upcoming internet of things era. While there have been some recent works [91] that attempt to solve the issue and establish the convergence guarantee of mean-field multi-agent RL, more investigation in this direction is still desired.…”

Machine Learning in Beyond 5G/6G Networks—State-of-the-Art and Future Trends

“…In [115], a multi-agent DRL-based framework was proposed for power control and maximization of throughput in energy-harvesting super IoT systems. Furthermore, a DNN based for distributed online power control is developed to study the policies in the system.…”

Reinforcement Learning Framework for Delay Sensitive Energy Harvesting Wireless Sensor Networks

“…In [28], the authors proposed a novel energy management algorithm to maximize the packet rate for point-to-point communication based on the actor-critic RL framework. In [29], a distributed multi-agent RL algorithm is proposed based on an identical reward function for all nodes.…”

“…Update θ (i) t and η (i) t using ( 28) and (29) t+1 has an associated potential function R t+1 (the centralized RL algorithm reward function) based on the potential games framework [41], [42]. Therefore, the optimization of the individual reward functions leads to a local optimal of the potential function under certain conditions.…”