DRL-Based Collaborative Edge Content Replication with Popularity Distillation

Abstract

The infrastructure for multimedia content delivery has been using more and more edge infrastructure (e.g., base stations, smart routers, etc.), which not only alleviates the centralized servers but also improves the quality of service by letting users access content nearby. Algorithms based on deep reinforcement learning (DRL) have been widely adopted by such edge cache replacement strategies due to their capability to adapt to changing request patterns. However, a DRL cache replacement agent learns extremely slow at an edge cache because of the sparse requests. In this paper, we propose a popularity distillation framework that allows edge caches to refer to content replication strategies of other edge caches. First, we design a collaborative edge cache framework that lets edge caches learn their strategies by handling the local requests using deep reinforcement learning and learn from others by exchanging the "soft" popularity distributions experienced by different edge caches. Second, we design a neighbor maintenance mechanism in which an agent iteratively selects only a small number of neighboring edge caches to perform the collaboration. Experiments driven by a real-world mobile video dataset show that our design can improve the cache hit rate by 3.0% compared with a non-popularity distillation baseline with only a small overhead of transmission data during distillation.

A Review on Software Defined Content Delivery Network: A Novel Combination of CDN and SDN

“…Takamasa H et al [97] try to solve the data loss issue by a micro cloudbased way that uploaded the content to different member devices. Haopeng Y et al [98] built a model of Inter-CDN based on the deep reinforcement learning and propose a replication algorithm based on popularity distribution.…”

LFDC: Low-Energy Federated Deep Reinforcement Learning for Caching Mechanism in Cloud–Edge Collaborative

“…First, each BS initializes MainNet and TargetNet based on its historical information and obtains the initial reply memory Ω (shown as lines 1-5). Next, DDQN is executed to train the caching process and update all parameters provided that there is no requested content f in BS n and the BS n storage is full (shown as lines [8][9][10][11][12][13][14][15][16][17]. Initialize replay memory Ω.…”

“…DRL, an AI model that can dynamically generate policies based on the environment, has seen successful applications in various cloud-edge collaborative environments, including wireless access network mode selection and resource management [9], autonomous driving strategy [10], and computation offloading optimization [11]. In the context of cache optimization, DRL methods have been utilized to improve cache efficiency by predicting content popularity [12] and dynamically adjusting cache policies [13][14][15]. Despite these achievements, using DRL-based caching strategies in cloud-edge environments still poses two significant challenges: (1) limited hardware resources, which leads to non-negligible energy consumption during DRL training, and (2) the need to jointly schedule cloud and edge storage resources for caching, which raises security and privacy concerns.…”

“…that is, to minimize the local loss function relative to the previous epoch (lines [10][11][12][13][14][15][16][17][18].…”