Deep Reinforcement Learning for Adaptive Caching in Hierarchical Content Delivery Networks

Abstract

Caching is envisioned to play a critical role in nextgeneration content delivery infrastructure, cellular networks, and Internet architectures. By smartly storing the most popular contents at the storage-enabled network entities during off-peak demand instances, caching can benefit both network infrastructure as well as end users, during on-peak periods. In this context, distributing the limited storage capacity across network entities calls for decentralized caching schemes. Many practical caching systems involve a parent caching node connected to multiple leaf nodes to serve user file requests. To model the two-way interactive influence between caching decisions at the parent and leaf nodes, a reinforcement learning (RL) framework is put forth. To handle the large continuous state space, a scalable deep RL approach is pursued. The novel approach relies on a hyper-deep Q-network to learn the Q-function, and thus the optimal caching policy, in an online fashion. Reinforcing the parent node with ability to learnand-adapt to unknown policies of leaf nodes as well as spatiotemporal dynamic evolution of file requests, results in remarkable caching performance, as corroborated through numerical tests.

Cognitive Caching at the Edges for Mobile Social Community Networks: A Multi-Agent Deep Reinforcement Learning Approach

“…However, [54] is mostly confined to centralized learning within one base station as a single agent which is not scalable in mobile edge-cloud scenarios [56]. Authors in [55] propose a deep reinforcement learning-based caching in hierarchical content delivery networks. The proposed framework DQNCache relies on Deep Q Networks to learn optimal caching policy in an online manner.…”

“…We model mobile edge-cloud network environment with a novel state space and set of actions that edges could take to collaborate with other nodes, interact with the environment in real time to maximise a cumulative reward. Traditional single-agent DRL-based caching approaches [54,55] have been proposed where a single edge (e.g. single base-station or an access-point) learns to make most suitable caching decisions based on the states of the environment and the rewards.…”

Data Caching at Fog Nodes Under IoT Networks: Review of Machine Learning Approaches

“…This approach increases the long term reward of the system and hence improves the performance. Deep Reinforcement Learning approach intelligently perceive the environment and automatically learns about the caching policy according to the history [68,69]. Emergence of deep neural network has made it feasible to automatically learn from raw and possibly high-dimensional data.…”

Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning

“…Deep neural networks (DNNs) can address the curse of dimensionality in the highdimensional and continuous state space by providing compact low-dimensional representations of high-dimensional inputs [19]. Wedding deep learning with RL (using a DNN to approximate the action-value function), deep (D) RL has offered artificial agents with human-level performance across diverse application domains [18], [20]. (D)RL algorithms have also shown great potential in several challenging power systems control and monitoring tasks [21], [22], [6], [23], [24], [25], and load control [26], [27].…”

“…Form the mini-batch loss L Tar (θ θ θ τ ; M τ ) using (19). 12: Update θ θ θ τ +1 using (20). 13: if mod(τ, B) = 0 then…”