Multi-Agent Deep Reinforcement Learning Based Spectrum Allocation for D2D Underlay Communications

Abstract

Device-to-device (D2D) communication underlay cellular networks is a promising technique to improve spectrum efficiency. In this situation, D2D transmission may cause severe interference to both the cellular and other D2D links, which imposes a great technical challenge to spectrum allocation. Existing centralized schemes require global information, which causes a large signaling overhead. While existing distributed schemes requires frequent information exchange among D2D users and cannot achieve global optimization. In this paper, a distributed spectrum allocation framework based on multiagent deep reinforcement learning is proposed, named multiagent actor critic (MAAC). MAAC shares global historical states, actions and policies during centralized training, requires no signal interaction during execution and utilizes cooperation among users to further optimize system performance. Moreover, in order to decrease the computing complexity of the training, we further propose the neighbor-agent actor critic (NAAC) based on the neighbor users' historical information for centralized training. The simulation results show that the proposed MAAC and NAAC can effectively reduce the outage probability of cellular links, greatly improve the sum rate of D2D links and converge quickly.Index Terms-Device-to-device (D2D) communications, multiagent deep reinforcement learning, spectrum allocation.

Distance-Based Resource Allocation for Vehicle-to-Pedestrian Safety Communication

“…Methods ranging from heuristic optimization [12] to machine learning [33], have been used under varying sets of users and their requirements. Machine learning, being the current hot topic in this domain; centralized vs. distributed reinforcement learning, multi-agent deep reinforcement learning [34], and other methods that could provide a faster convergence with better performance than traditional methods. Relay-assisted [35], out-of-coverage scenarios [36] and energy efficient resource allocation [37] is currently being researched upon.…”

Leveraging Machine-Learning for D2D Communications in 5G/Beyond 5G Networks

“…A survey of different popular and AI-based interference mitigation and RA approaches developed in D2D communications is provided in [24]. Additionally, the multi-agent actor critic (MAAC) is a newly proposed algorithm in [25] to mitigate interference by efficiently distributing the spectrum allocation. Moreover, the same paper proposes the neighbor-agent actor critic (NAAC) that uses neighbor users' historical information for centralized training leading to outage probability reduction and sum rate improvement for D2D links.…”

Deep Learning Based Power Allocation for Workload Driven Full-Duplex D2D-Aided Underlaying Networks

“…2) Deep learning based algorithms: Deep learning technology, which is based on deep neural network (DNN), has gained in popularity over the last decade due to its superior performance over the conventional techniques [34]- [40]. It is possible for us to solve complex non-linear problems in an efficient manner by using a back-propagation (BP) algorithm [41], in which a trained DNN model can be employed for reducing the computational time required in practical systems [37].…”