Deep Multi-Agent Reinforcement Learning Based Cooperative Edge Caching in Wireless Networks

Zhong, Chen; Gursoy, M. Cenk; Velipasalar, Senem

doi:10.1109/icc.2019.8762084

Cited by 44 publications

(20 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The real trace-driven evaluation results show that the proposed MAA2C reduces the latency around 21% and the cost around 26% compared with the state-of-art caching methods such as DRL [124], joint action learners [125]. The actor-critic framework for cooperative edge caching can also be found in [126]. However, unlike the unique model for each BS used in [123], the agent model of each BS in [126] contains a unique actor and a shared critic.…”

Section: B Content Cachingmentioning

confidence: 99%

Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey

Li¹,

Zhu²,

Luong³

et al. 2021

Preprint

View full text Add to dashboard Cite

Future Internet involves several emerging technologies such as 5G and beyond 5G networks, vehicular networks, unmanned aerial vehicle (UAV) networks, and Internet of Things (IoTs). Moreover, future Internet becomes heterogeneous and decentralized with a large number of involved network entities. Each entity may need to make its local decision to improve the network performance under dynamic and uncertain network environments. Standard learning algorithms such as single-agent Reinforcement Learning (RL) or Deep Reinforcement Learning (DRL) have been recently used to enable each network entity as an agent to learn an optimal decision-making policy adaptively through interacting with the unknown environments. However, such an algorithm fails to model the cooperations or competitions among network entities, and simply treats other entities as a part of the environment that may result in the non-stationarity issue. Multi-agent Reinforcement Learning (MARL) allows each network entity to learn its optimal policy by observing not only the environments, but also other entities' policies. As a result, MARL can significantly improve the learning efficiency of the network entities, and it has been recently used to solve various issues in the emerging networks. In this paper, we thus review the applications of MARL in the emerging networks. In particular, we provide a tutorial of MARL and a comprehensive survey of applications of MARL in next generation Internet. In particular, we first introduce single-agent RL and MARL. Then, we review a number of applications of MARL to solve emerging issues in future Internet. The issues consist of network access, transmit power control, computation offloading, content caching, packet routing, trajectory design for UAV-aided networks, and network security issues. Finally, we discuss the challenges, open issues, and future directions related to the applications of MARL in future Internet.

show abstract

Section: B Content Cachingmentioning

confidence: 99%

Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey

Li¹,

Zhu²,

Luong³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Example: Multi-agent task offloading [28] and multi-agent cooperative edge caching [29] are wireless problems which can be modeled as Dec-POMDP problems.…”

Section: ) Markov/stochastic Gamesmentioning

confidence: 99%

“…For edge caching, [29] and [104] propose MADDPG-like algorithms to solve the cooperative multi-agent edge caching problem. Both of these works model the cooperative edge caching as Dec-POMDP and differ in the definition of the state space and reward functions.…”

Section: Applications a Marl For Mec Systemsmentioning

confidence: 99%

“…Both of these works model the cooperative edge caching as Dec-POMDP and differ in the definition of the state space and reward functions. In [29], the edge servers receive the same reward as the average transmission delay reduction, whereas in [104], the weighted sum of the local and the neighbors' hit rates is considered as a reward signal to encourage cooperation between adjacent servers. Simulation results showed that the cooperative edge caching outperforms traditional caching mechanisms such as Least Recently Used (LRU), Least Frequently Used (LFU), and First In First Out (FIFO).…”

Section: Applications a Marl For Mec Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial

Feriani¹,

Hossain²

2020

Preprint

View full text Add to dashboard Cite

Deep Reinforcement Learning (DRL) has recently witnessed significant advances that have led to multiple successes in solving sequential decision-making problems in various domains, particularly in wireless communications. The future sixth-generation (6G) networks are expected to provide scalable, low-latency, ultra-reliable services empowered by the application of data-driven Artificial Intelligence (AI). The key enabling technologies of future 6G networks, such as intelligent metasurfaces, aerial networks, and AI at the edge, involve more than one agent which motivates the importance of multi-agent learning techniques. Furthermore, cooperation is central to establishing self-organizing, self-sustaining, and decentralized networks. In this context, this tutorial focuses on the role of DRL with an emphasis on deep Multi-Agent Reinforcement Learning (MARL) for AI-enabled 6G networks. The first part of this paper will present a clear overview of the mathematical frameworks for single-agent RL and MARL. The main idea of this work is to motivate the application of RL beyond the model-free perspective which was extensively adopted in recent years. Thus, we provide a selective description of RL algorithms such as Model-Based RL (MBRL) and cooperative MARL and we highlight their potential applications in 6G wireless networks. Finally, we overview the state-of-the-art of MARL in fields such as Mobile Edge Computing (MEC), Unmanned Aerial Vehicles (UAV) networks, and cell-free massive MIMO, and identify promising future research directions. We expect this tutorial to stimulate more research endeavors to build scalable and decentralized systems based on MARL.

show abstract

“…Age of information has been proposed as an effective metric to quantify the freshness of information in communication networks. There have been lots of efforts on Age of information such as social networks [7], web crawling [8]- [10], queueing networks [11]- [14], caching systems [15]- [24], scheduling in networks [25]- [28], multi-hop multicast networks [29]- [31], reinforcement learning [32]- [33] and so on.…”

Section: Introductionmentioning

confidence: 99%

Freshness-Optimal Caching for Information Updating Systems with Limited Cache Storage Capacity

Xie,

Cheng,

Zhang

2020

Preprint

View full text Add to dashboard Cite

In this paper, we investigate a cache updating system with a server containing N files, K relays and M users. The server keeps the freshest versions of the files which are updated with fixed rates. Each relay can download the fresh files from the server in a certain period of time. Each user can get the fresh files from any relay as long as the relay has stored the fresh versions of the requested files. Due to the limited storage capacity and updating capacity of each relay, different cache designs will lead to different average freshness of all updating files at users. In order to keep the average freshness as large as possible in the cache updating system, we formulate an average freshness-optimal cache updating problem (AFOCUP) to obtain an optimal cache scheme. However, because of the nonlinearity of the AFOCUP, it is difficult to seek out the optimal cache scheme. As a result, an linear approximate model is suggested by distributing the total update rates completely in accordance with the number of files in the relay in advance. Then we utilize the greedy algorithm to search the optimal cache scheme that is satisfied with the limited storage capacity of each relay. Finally, some numerical examples are provided to illustrate the performance of the approximate solution.

show abstract

Deep Multi-Agent Reinforcement Learning Based Cooperative Edge Caching in Wireless Networks

Cited by 44 publications

References 14 publications

Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey

Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey

Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial

Freshness-Optimal Caching for Information Updating Systems with Limited Cache Storage Capacity

Contact Info

Product

Resources

About