A Deep Reinforcement Learning Framework for Contention-Based Spectrum Sharing

Doshi, Akash; Yerramalli, Srinivas; Ferrari, Lorenzo; Yoo, Taesang; Andrews, Jeffrey G.

doi:10.1109/jsac.2021.3087254

Cited by 33 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…BSs. This is a significant improvement in scalability over [20] which had only 4 BSs, as well as other recent works utilizing multi-agent RL for cognitive radio [18], [19] which utilize under ten BSs and only perform adaptive medium access, not modulation. The rapid training convergence, low training sample complexity and the significant performance improvement over genie-aided non-ML baselines can be attributed to the decentralized actor centralized critic approach we employ to learn a RL policy at each BS.…”

Section: Contributionsmentioning

confidence: 81%

“…None of these papers thus far have attempted to model the asynchronous nature of the decisions made by the transmitters owing to contention. In [20], we developed a distributed deep RL spectrum sharing algorithm incorporating contention-based medium access. It deployed Deep Q Networks (DQN) at each BS that sequentially decide whether or not to transmit, with the goal of maximizing proportional fairness (PF) network-wide.…”

Section: B Related Workmentioning

confidence: 99%

“…It deployed Deep Q Networks (DQN) at each BS that sequentially decide whether or not to transmit, with the goal of maximizing proportional fairness (PF) network-wide. In this paper, we provide a nontrivial generalization of [20], by expanding the action space of the BS from a simple yes/no transmit decision to a choice of modulation scheme, hence providing a framework for rate adaptation. Moreover, we propose modifications to the training algorithm, including the development of a new policy gradient based approach, that significantly improve the maximization of the proportional fairness metric from [20] and provide for scalability of the approach to more practical scenarios with a large number of BSs.…”

Section: B Related Workmentioning

confidence: 99%

“…In [20], given that a BS will only have delayed access to parameters of the UE it serves and must operate in a decentralized fashion, we formulated medium access decisions as a decentralized partially observable Markov decision process (DEC-POMDP) and subsequently adapted it to contention. We provide a brief overview of the aforementioned formulation in this section, with the notation adapted for adaptive modulation.…”

Section: Contention-based Medium Access As a Dec-pomdpmentioning

confidence: 99%

“…). Note that the only key changes from ( 23) are that we have added a term for training V EOS To overcome this partial observability, an LSTM layer is introduced in V CON i , V EOS i and π CON i at every BS i as will be explained in Section IV-E. Now, to compute V target,CON i and V target,EOS i , we first observe that (20), when applied to the EOS-CON transition yields…”

Section: Proximal Policy Optimization (Ppo)mentioning

confidence: 99%

See 4 more Smart Citations

Distributed Deep Reinforcement Learning for Adaptive Medium Access and Modulation in Shared Spectrum

Doshi¹,

Andrews²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Spectrum scarcity has led to growth in the use of unlicensed spectrum for cellular systems. This motivates intelligent adaptive approaches to spectrum access for both WiFi and 5G that improve upon traditional carrier sensing and listen-before-talk methods. We study decentralized contention-based medium access for base stations (BSs) of a single Radio Access Technology (RAT) operating on unlicensed shared spectrum. We devise a learning-based algorithm for both contention and adaptive modulation that attempts to maximize a network-wide downlink throughput objective. We formulate and develop novel distributed implementations of two deep reinforcement learning approaches -Deep Q Networks and Proximal Policy Optimization -modelled on a two stage Markov decision process.Empirically, we find the (proportional fairness) reward accumulated by the policy gradient approach to be significantly higher than even a genie-aided adaptive energy detection threshold. Our approaches are further validated by improved sum and peak throughput. The scalability of our approach to large networks is demonstrated via an improved cumulative reward earned on both indoor and outdoor layouts with a large number of BSs.

show abstract

Section: Contributionsmentioning

confidence: 81%

Section: B Related Workmentioning

confidence: 99%

Section: B Related Workmentioning

confidence: 99%

Section: Contention-based Medium Access As a Dec-pomdpmentioning

confidence: 99%

Section: Proximal Policy Optimization (Ppo)mentioning

confidence: 99%

See 3 more Smart Citations

Distributed Deep Reinforcement Learning for Adaptive Medium Access and Modulation in Shared Spectrum

Doshi¹,

Andrews²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Machine learning for cooperative spectrum sensing and sharing: A survey

Janu

Singh

Kumar

2021

Trans Emerging Tel Tech

View full text Add to dashboard Cite

With the rapid development of next-generation wireless communication technologies and the increasing demand of spectrum resources, it becomes necessary to introduce learning and reasoning capabilities in cognitive radio networks (CRN). In particular, our focus is on two fundamental applications in CRNs, namely spectrum sensing (SS) and spectrum sharing. The application of machine learning (ML) techniques has added new aspects to SS and spectrum sharing. This paper offers a survey on various ML-based algorithms in the cooperative spectrum sensing (CSS) and dynamic spectrum sharing (DSS) domain, with its emphasis on types of features extracted from primary user signal, types of ML algorithm, and performance metrics utilized for evaluation of ML algorithms. Starting with the basic principles and challenges of SS, this paper also justifies the applicability of supervised, unsupervised, and reinforcement ML algorithms in the CSS domain. The application of ML algorithms, to solve the DSS problem has also been reviewed. Finally, the survey paper is concluded with some suggested open research challenges and future directions for ML application in next-generation communication technologies.

show abstract