A Deep Actor-Critic Reinforcement Learning Framework for Dynamic Multichannel Access

Abstract

To make efficient use of limited spectral resources, we in this work propose a deep actor-critic reinforcement learning based framework for dynamic multichannel access. We consider both a single-user case and a scenario in which multiple users attempt to access channels simultaneously. We employ the proposed framework as a single agent in the single-user case, and extend it to a decentralized multi-agent framework in the multi-user scenario. In both cases, we develop algorithms for the actor-critic deep reinforcement learning and evaluate the proposed learning policies via experiments and numerical results. In the single-user model, in order to evaluate the performance of the proposed channel access policy and the framework's tolerance against uncertainty, we explore different channel switching patterns and different switching probabilities. In the case of multiple users, we analyze the probabilities of each user accessing channels with favorable channel conditions and the probability of collision. We also address a time-varying environment to identify the adaptive ability of the proposed framework.Additionally, we provide comparisons (in terms of both the average reward and time efficiency) between the proposed actor-critic deep reinforcement learning framework, Deep-Q network (DQN) based approach, random access, and the optimal policy when the channel dynamics are known.

Deep Reinforcement Learning for Dynamic Spectrum Sensing and Aggregation in Multi-Channel Wireless Networks

“…Recently DRL has achieved significant breakthroughs in the dynamic spectrum allocation problems [27]- [35]. The works in [27], [28], [29], [30] and [31] studied the multichannel access problem under the assumption of Markov spectrum occupancy model. The authors of [27] considered the highly correlation between channels thus the user can access the vacant channel by historical partial observations.…”

“…The authors of [27] considered the highly correlation between channels thus the user can access the vacant channel by historical partial observations. A actorcritic DRL based framework was proposed in [28], [29] and its performance was further improved in [27] especially in scenarios with a large number of channels. In [30] all channels are independent so the user is supposed to have fully observation of the system via wideband spectrum sensing techniques.…”

“…The independent channel model is also adopted in [31], but the authors of [31] considered the presence of spectrum sensing errors, and the position of each user is specified in the proposed scenario. Moreover, multi-user scenarios are also studied in [29] and [31] through distributive learning. However, in the most of aforementioned works the user only selects one channel to access with the hope of avoiding collisions at each time slot.…”

“…However, in the most of aforementioned works the user only selects one channel to access with the hope of avoiding collisions at each time slot. The authors of [29] considered a scenario where the user can access more than one channel at a time, but it has nothing to do with the user's requirement. In other words, previous works mainly focused on allocating the single channel to the user without taking into account the user's demand of bandwidth.…”

“…The multichannel access problem has been studied in [27]- [31] with single user where the state transition of channels is modeled as a joint Markov chain. The correlation between channels has been taken into account in [27]- [29] while an independent channel model has been used in [30], [31]. The efficiency of spectrum aggregation and the spectrum assignment problem has been studied in [6] and [7], respectively.…”