Deep Reinforcement Learning for Dynamic Spectrum Access in Wireless Networks

Xu, Yue; Yu, Jiabin; Headley, William C.; Buehrer, R. Michael

doi:10.1109/milcom.2018.8599723

Cited by 45 publications

(21 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apart from [12], there have also been other work on DRL based MAC. As in [12], investigations in [16][17][18][19][20][21] also assumed the physical channels are perfect and did not have a scheme to deal with imperfect channels. Other fine differences between [16][17][18][19][20][21] and our current work are elaborated in the following.…”

Section: A Related Workmentioning

confidence: 99%

“…The difference is that in [16], the authors assumed the channels are time-invariant, whereas the channels in [17][18][19][20] may be time-varying in that some primary or "legacy" users may occupy the channels from time to time. Therefore, the wireless networks in [17][18][19][20] can also be regarded as "heterogeneous". However, the DRL users in [17][18][19][20] aim to maximize their own throughputs by learning the channel characteristics and the transmission patterns of the "primary" or "legacy" users.…”

Section: A Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks

Wang

Liew

2019

IEEE J. Select. Areas Commun.

298

181

View full text Add to dashboard Cite

This paper investigates a futuristic spectrum sharing paradigm for heterogeneous wireless networks with imperfect channels. In the heterogeneous networks, multiple wireless networks adopt different medium access control (MAC) protocols to share a common wireless spectrum and each network is unaware of the MACs of others. This paper aims to design a distributed deep reinforcement learning (DRL) based MAC protocol for a particular network, and the objective of this network is to achieve a global α-fairness objective. In the conventional DRL framework, feedback/reward given to the agent is always correctly received, so that the agent can optimize its strategy based on the received reward. In our wireless application where the channels are noisy, the feedback/reward (i.e., the ACK packet) may be lost due to channel noise and interference. Without correct feedback, the agent (i.e., the network user) may fail to find a good solution. Moreover, in the distributed protocol, each agent makes decisions on its own. It is a challenge to guarantee that the multiple agents will make coherent decisions and work together to achieve the same objective, particularly in the face of imperfect feedback channels. To tackle the challenge, we put forth (i) a feedback recovery mechanism to recover missing feedback information, and (ii) a two-stage action selection mechanism to aid coherent decision making to reduce transmission collisions among the agents. Extensive simulation results demonstrate the effectiveness of these two mechanisms. Last but not least, we believe that the feedback recovery mechanism and the two-stage action selection mechanism can also be used in general distributed multi-agent reinforcement learning problems in which feedback information on rewards can be corrupted.

show abstract

Section: A Related Workmentioning

confidence: 99%

Section: A Related Workmentioning

confidence: 99%

Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks

Wang

Liew

2019

IEEE J. Select. Areas Commun.

298

181

View full text Add to dashboard Cite

show abstract

“…Since this paper focuses on MAC designs based on DRL techniques, we limit our review of related work in the same area only. A substantial body of past related work on DRL based MAC, for example, [17][18][19][20][21], did not consider MAC with carrier sensing. In the set-up of the past work, the time steps in decision making are with the same duration.…”

Section: B Related Workmentioning

confidence: 99%

“…In the following, we elaborate other fine differences between our work and [17][18][19][20][21][22][23]. The DRL MAC proposed in [17] is targeted for homogeneous wireless networks.…”

Section: B Related Workmentioning

confidence: 99%

Non-Uniform Time-Step Deep Q-Network for Carrier-Sense Multiple Access in Heterogeneous Wireless Networks

Liew

Wang

2021

IEEE Trans. on Mobile Comput.

View full text Add to dashboard Cite

This paper investigates a new class of carrier-sense multiple access (CSMA) protocols that employ deep reinforcement learning (DRL) techniques, referred to as carrier-sense deep-reinforcement learning multiple access (CS-DLMA). The goal of CS-DLMA is to enable efficient and equitable spectrum sharing among a group of co-located heterogeneous wireless networks. Existing CSMA protocols, such as the medium access control (MAC) of WiFi, are designed for a homogeneous network in which all nodes adopt the same protocol. Such protocols suffer from severe performance degradation in a heterogeneous environment where there are nodes adopting other MAC protocols. CS-DLMA aims to circumvent this problem by making use of DRL. In particular, this paper adopts α-fairness as the general objective of CS-DLMA. With α-fairness, CS-DLMA can achieve a range of different objectives (e.g., maximizing sum throughput, achieving proportional fairness, or achieving maxmin fairness) when coexisting with other MACs by changing the value of α. A salient feature of CS-DLMA is that it can achieve these objectives without knowing the coexisting MACs through a learning process based on DRL. The underpinning DRL technique in CS-DLMA is deep Q-network (DQN). However, the conventional DQN algorithms are not suitable for CS-DLMA due to their uniform time-step assumption. In CSMA protocols, time steps are non-uniform in that the time duration required for carrier sensing is smaller than the duration of data transmission. This paper introduces a non-uniform time-step formulation of DQN to address this issue. Our simulation results show that CS-DLMA can achieve the general α-fairness objective when coexisting with TDMA, ALOHA, and WiFi protocols by adjusting its own transmission strategy. Interestingly, we also find that CS-DLMA is more Pareto efficient than other CSMA protocols, e.g., p-persistent CSMA, when coexisting with WiFi. Although this paper focuses on the use of our non-uniform time-step DQN formulation in wireless networking, we believe this new DQN formulation can also find use in other domains.

show abstract

“…Based on enough SI, the SK is obtained through the data mining and knowledge construction. The form of the SK can be the spectrum prediction model, the spectrum management paradigms [24], spectrum access strategy [35], etc. Compared with the SI, the SK can be more directly used for the SOP exploration and exploitation.…”

Section: ) Layer 3: Skmentioning

confidence: 99%

Spectrum Knowledge and Real-Time Observing Enabled Smart Spectrum Management

et al. 2020

View full text Add to dashboard Cite

Spectrum is the indispensable resource for 5G wireless systems and beyond. The dynamic spectrum management (DSM) framework with spectrum sharing among different kinds of users is essential to satisfy the sustainable rapid increase of the bandwidth requirement. Based on the analysis of the DSM as well as the latent applications of the machine learning and artificial intelligence (AI) in the spectrum access, this paper will address the problem of incorporating intelligence into the spectrum management. Firstly, the spectrum opportunity (SOP) related information is layered into the spectrum data, the spectrum information and the spectrum knowledge based on the processing and abstractive level. The spectrum knowledge is formally defined as the extendible and scalable information to reason and predict the SOP usability as well as the outcome of the SOP occupancy. Then the smart spectrum model (SSM) is proposed with the spectrum knowledge as the key enabler and the coupled SOP exploration and exploitation as the core feature. Under the SSM framework, the spectrum knowledge and real-time observing (SKRO) enabled SOP exploration scheme is designed, which can make use of both the historical information and the real-time sensing information for smart SOP exploration. Extensive simulations are provided which demonstrate that the SKRO enabled SSM can achieve much better SOP utilization while satisfying the required legacy assurance. Specifically, in the low signal to noise environment, at least 16.55% gains on the SOP utilization ratio can be obtained compared with the DSM.

show abstract

Deep Reinforcement Learning for Dynamic Spectrum Access in Wireless Networks

Cited by 45 publications

References 6 publications

Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks

Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks

Non-Uniform Time-Step Deep Q-Network for Carrier-Sense Multiple Access in Heterogeneous Wireless Networks

Spectrum Knowledge and Real-Time Observing Enabled Smart Spectrum Management

Contact Info

Product

Resources

About