A Survey on Applications of Model-Free Strategy Learning in Cognitive Wireless Networks

Abstract

Abstract-The framework of cognitive wireless radio is expected to endow the wireless devices with the cognitionintelligence ability, with which they can efficiently learn and respond to the dynamic wireless environment. In many practical scenarios, the complexity of network dynamics makes it difficult to determine the network evolution model in advance. As a result, the wireless decision-making entities may face a black-box network control problem and the model-based network management mechanisms will be no longer applicable. In contrast, model-free learning has been considered as an efficient tool for designing control mechanisms when the model of the system environment or the interaction between the decision-making entities is not available as a-priori knowledge. With model-free learning, the decision-making entities adapt their behaviors based on the reinforcement from their interaction with the environment and are able to (implicitly) build the understanding of the system through trial-and-error mechanisms. Such characteristics of model-free learning is highly in accordance with the requirement of cognition-based intelligence for devices in cognitive wireless networks. Recently, model-free learning has been considered as one key implementation approach to adaptive, self-organized network control in cognitive wireless networks. In this paper, we provide a comprehensive survey on the applications of the stateof-the-art model-free learning mechanisms in cognitive wireless networks. According to the system models that those applications are based on, a systematic overview of the learning algorithms in the domains of single-agent system, multi-agent systems and multi-player games is provided. Furthermore, the applications of model-free learning to various problems in cognitive wireless networks are discussed with the focus on how the learning mechanisms help to provide the solutions to these problems and improve the network performance over the existing model-based, non-adaptive methods. Finally, a broad spectrum of challenges and open issues is discussed to offer a guideline for the future research directions.

Interference Constraint Active Learning with Uncertain Feedback for Cognitive Radio Networks

“…In [13], a binary Spectrum Sensing feedback has been used to enable CRs apply a Reinforcement Learning procedure, the Q-Learning, to regulate the aggregated interference to the PU and in [14] a distributed channel admission solution is formulated based on Dynamic Programming, while in their previous work [15] a SU power control policy is also included using the same formulation but without elaborating on the belief factor enhancement. In [16], a methodical overview of all the Reinforcement Learning applications in CRNs based on the Markov Decision Process framework is provided. Additionally, the authors of [6] proposed a CPM based learning algorithm where probing the PU system targets to both learning interference channel matrices and maximizing the SNR at the SU receiver side in an underlay cognitive BF scenario.…”

Application of Machine Learning in Wireless Networks: Key Techniques and Open Issues

“…In [15], different learning techniques, which are suitable for Internet of Things (IoT), are presented, taking into account the unique characteristics of IoT, including resource constraints and strict quality-of-service requirements, and studies on learning for IoT are also reviewed in [16]. The applications of machine learning in cognitive radio (CR) environments are investigated in [17] and [18]. Specifically, authors in [17] classify those applications into decisionmaking tasks and classification tasks, while authors in [18] mainly concentrate on model-free strategic learning.…”

Performance Analysis of Trial and Error Algorithms

“…In practice, the optimization/game frameworks can however very complex, in which the utility function may not have a closed form expression and even may take discrete values. In such contexts, model-free strategy learning algorithms are very appealing approaches [3]. Players neither try to model the environment nor try to have a specific/explicit utility form.…”

“…In model-free resource allocation schemes, developing decentralized strategies that converge to an equilibrium (if it exists), or at least finding conditions under which they converge, represent a main challenge [3]. The trial and error algorithms, proposed in [4], [12] and then applied to various resource sharing problems e.g.…”

“…On the other hand, Trial and Error Learning (TEL) algorithm from [12] has the property to spend a high proportion of time in an optimum state that maximizes the sum of utility of all players if there is a PNE, otherwise, it spends a high proportion of time in an optimum state that maximizes a tradeoff between the sum of utility of all players and a predefined stability function. Even though the above two algorithms converge to a desired state, the convergence rate remains an open question [3], [12]. The main reason comes from the computation complexity of the inherent Markov Chain (MC) generated by these two algorithms.…”