Multi-Agent Reinforcement Learning for Cooperative Coded Caching via Homotopy Optimization

Abstract

Introducing cooperative coded caching into small cell networks is a promising approach to reducing traffic loads. By encoding content via maximum distance separable (MDS) codes, coded fragments can be collectively cached at small-cell base stations (SBSs) to enhance caching efficiency. However, content popularity is usually time-varying and unknown in practice. As a result, cached content is anticipated to be intelligently updated by taking into account limited caching storage and interactive impacts among SBSs. In response to these challenges, we propose a multi-agent deep reinforcement learning (DRL) framework to intelligently update cached content in dynamic environments. With the goal of minimizing long-term expected fronthaul traffic loads, we first model dynamic coded caching as a cooperative multi-agent Markov decision process. Owing to the use of MDS coding, the resulting decision-making falls into a class of constrained reinforcement learning problems with continuous decision variables. To deal with this difficulty, we custom-build a novel DRL algorithm by embedding homotopy optimization into a deep deterministic policy gradient formalism. Next, to empower the caching framework with an effective trade-off between complexity and performance, we propose centralized, and partially and fully decentralized caching controls by applying the derived DRL approach. Simulation results demonstrate the superior performance of the proposed multi-agent framework.

An Interactive Self-Learning Game and Evolutionary Approach Based on Non-Cooperative Equilibrium

“…The goal of the critic network is to minimize the loss L θ Q , since a smaller L θ Q represents a more accurate value given for states and actions. Therefore, the parameter θ Q of the evaluation network is updated by the gradient descent method, and the equation is shown as (4).…”

“…where A t = π θ u (S t ), and γ is the discount factor. The learning objective of the critic network is to minimize L θ Q , and the parameters θ Q of the critic network are updated by the gradient descent method, and the formula is shown in Equation (4).…”

“…In recent years, deep reinforcement learning (DRL) has gradually become a research focus and the development trend in the field of artificial intelligence and was selected by MIT as one of the 10 breakthrough technologies in 2017 [1]. Various deep learning methods such as Dagger [2], Deep Q-learning [3], DRL [4] and DNQ [5] are publicized. Among them, the Deep Deterministic Policy Gradient (DDPG) is widely used because of its excellent ability to observe and execute actions instantly in terms of individual intelligence [6], such as for the robotic arms to achieve high precise actions [7], for the Autonomous Underwater Vehicles (AUVs) to patrol intelligently [8].…”

A Survey on Reinforcement Learning-Aided Caching in Heterogeneous Mobile Edge Networks

“…In [130], maximum distance separable codes are used for cooperative coded caching at small BSs in a scenario with time-varying and unknown content popularity. Aiming to address this challenge and guarantee reduced fronthaul load, a multi-agent DRL solution is presented for cache update.…”

Learning from Peers: Deep Transfer Reinforcement Learning for Joint Radio and Cache Resource Allocation in 5G RAN Slicing

“…[17] also jointly considers the content caching and mode selection problem, which applies DRL for decision making of the cloud server to satisfy the requirements of slices. The multi-tenant radio resource allocation problem is investigated in [18], and DRL is leveraged to optimize the computation offloading and packet scheduling. A multi-agent DRL algorithm is proposed in [19] for the cooperative content caching of small base stations, and each agent can intelligently cache contents under time-varying popularity.…”