Reinforcement Learning for Adaptive Resource Allocation in Fog RAN for IoT With Heterogeneous Latency Requirements

Abstract

In light of the quick proliferation of Internet of things (IoT) devices and applications, fog radio access network (Fog-RAN) has been recently proposed for fifth generation (5G) wireless communications to assure the requirements of ultra-reliable low-latency communication (URLLC) for the IoT applications which cannot accommodate large delays. Hence, fog nodes (FNs) are equipped with computing, signal processing and storage capabilities to extend the inherent operations and services of the cloud to the edge. We consider the problem of sequentially allocating the FN's limited resources to the IoT applications of heterogeneous latency requirements. For each access request from an IoT user, the FN needs to decide whether to serve it locally utilizing its own resources or to refer it to the cloud to conserve its valuable resources for future users of potentially higher utility to the system (i.e., lower latency requirement). We formulate the Fog-RAN resource allocation problem in the form of a Markov decision process (MDP), and employ several reinforcement learning (RL) methods, namely Q-learning, SARSA, Expected SARSA, and Monte Carlo, for solving the MDP problem by learning the optimum decision-making policies. We verify the performance and adaptivity of the RL methods and compare it with the performance of a fixed-thresholdbased algorithm. Extensive simulation results considering 19 IoT environments of heterogeneous latency requirements corroborate that RL methods always achieve the best possible performance regardless of the IoT environment.

Reinforcement Learning-Based Resource Management Model for Fog Radio Access Network Architectures in 5G

“…The proposed reinforcement learning framework in this paper implementing a proactive auto-scaling algorithm based on Q-learning will be referred to as System I. The work in [34] described several RL-based methods for resource allocation in FRAN architectures. The algorithm based on SARSA will be referred to as System II, while the Monte Carlo mechanism is System III.…”

Home Energy Recommendation System (HERS): A Deep Reinforcement Learning Method for Electricity Optimization in Smart Homes

“…Reinforcement learning (RL) techniques are typically preferred for their datadriven online decision-making capability. Recent advances in neural network based deep RL algorithms lead to widespread applications, including gaming [12], finance [13], transportation [14], communications [15], environmental systems [16], and healthcare systems [17]. [18] provides an extensive review of RL for DSM, showcasing the suitability of RL for DSM techniques.…”

A Review of Techniques and Methods for IoT Applications in Collaborative Cloud-Fog Environment

“…Nassar and Yilmaz [28] formulated the resource allocation problem as a Markov decision process and finally solved it by learning the optimal decision strategy with some reinforcement learning methods, namely, SARSA, Expected SARSA, Q-learning, and Monte Carlo. eir work can make the fog node decide the processing location of service request according to its own resources, realizing the low latency transaction offloading and processing with high performance.…”

“…Combines the computation offloading and sleep decisions of node servers to maximize server quality and reduces energy consumption [28] An optimal decision strategy with some reinforcement learning methods…”