Federated Multiagent Actor–Critic Learning for Age Sensitive Mobile-Edge Computing

Zhu, Zheqi; Wan, Shuo; Fan, Pingyi; Letaief, Khaled B.

doi:10.1109/jiot.2021.3078514

Cited by 97 publications

(42 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…searching algorithms [79], [85], [86], heuristic algorithms [87], and game theory [88], [89]. Alternatively, integrating RL and machine learning algorithms is adopted, such as supervised/unsupervised learning [90]- [92], federated learning (FL) [93]- [95], and hierarchical RL [64], [96]- [99], for optimization efficiency improvement. In large-scale networks, the first solution may not be able to make the best policy of decision-making and resource allocation.…”

Section: B Reinforcement Learning-empowered Mecmentioning

confidence: 99%

“…2) Mobile MEC Server: To satisfy the extensive service requests of a tremendous number of mobile devices, vehicle-and UAV-aided network architectures have been proposed with mobile MEC servers [60]- [64], [76], [88], [95], [96], [133]- [139]. Due to the flexible coverage of the movable MEC servers, the computational service range is sufficiently extended.…”

Section: Studies On Distributed Service 1) Complex Service Deploymentmentioning

confidence: 99%

“…In addition, coexisting ground vehicles and UAVs will create more complicated network dynamics in MEC. In [95], a joint optimization problem of trajectory planning, task scheduling, and resource allocation was investigated for system timeliness, i.e., the freshness of data and computational tasks. The concept of AoI in [56] is used to evaluate the content freshness, which is expressed as the time of the current content elapsed since it is generated.…”

Section: Studies On Distributed Service 1) Complex Service Deploymentmentioning

confidence: 99%

See 2 more Smart Citations

Reinforcement Learning-Empowered Mobile Edge Computing for 6G Edge Intelligence

Wei,

Guo,

et al. 2022

Preprint

View full text Add to dashboard Cite

Mobile edge computing (MEC) is considered a novel paradigm for computation-intensive and delay-sensitive tasks in fifth generation (5G) networks and beyond. However, its uncertainty, referred to as dynamic and randomness, from the mobile device, wireless channel, and edge network sides, results in high-dimensional, nonconvex, nonlinear, and NP-hard optimization problems. Thanks to the evolved reinforcement learning (RL), upon iteratively interacting with the dynamic and random environment, its trained agent can intelligently obtain the optimal policy in MEC. Furthermore, its evolved versions, such as deep RL (DRL), can achieve higher convergence speed efficiency and learning accuracy based on the parametric approximation for the large-scale state-action space. This paper provides a comprehensive research review on RL-enabled MEC and offers insight for development in this area. More importantly, associated with free mobility, dynamic channels, and distributed services, the MEC challenges that can be solved by different kinds of RL algorithms are identified, followed by how they can be solved by RL solutions in diverse mobile applications. Finally, the open challenges are discussed to provide helpful guidance for future research in RL training and learning MEC. Index Terms-Mobile edge computing (MEC), network uncertainty, reinforcement learning (RL). emote Areas irplane taverse vers Diverse Scenarios MEC Deployment Future Applications Smart City Smart Home Hazard/Remote Areas Smart Agriculture Digital Twin Holographic Communication AR/VR/XR Metaverse Metaverse Metavers Autonomous Driving Small Data Center Switch BS Traffic Light Satellite Ship Airplane S ll S it h BS T ffi Li ht S t llit Shi Ai l 5G Networks and Beyond 5G Networks and Beyond Smart Factory performance. Its typical algorithms are K-means clustering, principal component analysis, and independent component analysis, which can be used in small cell clustering, heterogeneous network clustering, smart grid user classification, etc. However, in unreliable radio network environments, the classification accuracy decreases, which readily causes slow and inaccurate actions in MEC systems. • Different from the static solutions provided by supervised learning and unsupervised learning, RL gives a constantly evolving intelligent framework [21]-[27]. In RL [29], an agent is enabled to make proper decisions based on frequent interactions with the stochastic and dynamic environment. Based on the Markov models, the RL operates in a feedback mechanism (a closed loop) without the knowledge of input data, where based on the previous and current states, the agent can execute its actions to maximize the reward function. Additionally, the above behaviors, including the states, actions, and rewards, accumulate to generate experiences. The classic RL algorithm is Q-learning, which suffers from the curse of dimensionality caused by the large dimensions of the state-action spaces. Accordingly, upon leveraging the low-dimensional representation for the high-dimensional state-ac...

show abstract

Section: B Reinforcement Learning-empowered Mecmentioning

confidence: 99%

Section: Studies On Distributed Service 1) Complex Service Deploymentmentioning

confidence: 99%

Section: Studies On Distributed Service 1) Complex Service Deploymentmentioning

confidence: 99%

See 1 more Smart Citation

Reinforcement Learning-Empowered Mobile Edge Computing for 6G Edge Intelligence

Wei,

Guo,

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Apart from caching services, FRL has demonstrated its strong ability to facilitate resource allocation in edge computing. In [93], the authors specifically focus on the data offloading task for Mobile Edge Computing (MEC) systems. To achieve joint collaboration, the heterogeneous multi-agent actor-critic (H-MAAC) framework is proposed, in which edge devices independently learn the interactive strategies through their own observations.…”

Section: Frl For Edge Computingmentioning

confidence: 99%

“…Among the applications of FRL, most researchers focus on the communication network system due to its robust security requirements, advanced distributed architecture, and a variety of decision-making tasks. Data offloading [93] and caching [89] solutions powered by distributed AI are available from FRL. In addition, with the ability to detect a wide range of attacks and support defense solutions, FRL has emerged as a strong alternative for performing distributed learning for security-sensitive scenarios.…”

Section: Lessons Learned From the Relationship Between Fl And Rlmentioning

confidence: 99%

Federated reinforcement learning: techniques, applications, and open challenges

Qi¹,

Zhou²,

Lei³

et al. 2021

View full text Add to dashboard Cite

This paper presents a comprehensive survey of Federated Reinforcement Learning (FRL), an emerging and promising field in Reinforcement Learning (RL). Starting with a tutorial of Federated Learning (FL) and RL, we then focus on the introduction of FRL as a new method with great potential by leveraging the basic idea of FL to improve the performance of RL while preserving data-privacy. According to the distribution characteristics of the agents in the framework, FRL algorithms can be divided into two categories, i.e., Horizontal Federated Reinforcement Learning (HFRL) and Vertical Federated Reinforcement Learning (VFRL). We provide the detailed definitions of each category by formulas, investigate the evolution of FRL from a technical perspective, and highlight its advantages over previous RL algorithms. In addition, the existing works on FRL are summarized by application fields, including edge computing, communication, control optimization, and attack detection. Finally, we describe and discuss several key research directions that are crucial to solving the open problems within FRL.

show abstract

Toward intelligent resource management in dynamic Fog Computing‐based Internet of Things environment with Deep Reinforcement Learning: A survey

Gupta

Singh

2022

Int J Communication

View full text Add to dashboard Cite

Fog computing has already started to gain a lot of momentum in the industry for its ability to turn scattered computing resources into a large-scale, virtualized, and elastic computing environment. Resource management (RM) is one of the key challenges in fog computing which is also related to the success of fog computing. Deep learning has been applied to the fog computing field for some time, and it is widely used in large-scale network RM. Reinforcement learning (RL) is a type of machine learning algorithms, and it can be used to learn and make decisions based on reward signals that are obtained from interactions with the environment. We examine current research in this area, comparing RL and deep reinforcement learning (DRL) approaches with traditional algorithmic methods such as graph theory, heuristics, and greedy for managing resources in fog computing environments (published between 2013 and 2022) illustrating how RL and DRL algorithms can be more effective than conventional techniques. Various algorithms based on DRL has been shown to be applicable to RM problem and proved that it has a lot of potential in fog computing. A new microservice model based on the DRL framework is proposed to achieve the goal of efficient fog computing RM. The positive impact of this work is that it can successfully provide a resource manager to efficiently schedule resources and maximize the overall performance. K E Y W O R D S AoI (age of information), deep reinforcement learning (DRL), fog computing (FC), Internet of Things (IoT), Markov decision process, Q-learning

show abstract

Federated Multiagent Actor–Critic Learning for Age Sensitive Mobile-Edge Computing

Cited by 97 publications

References 55 publications

Reinforcement Learning-Empowered Mobile Edge Computing for 6G Edge Intelligence

Reinforcement Learning-Empowered Mobile Edge Computing for 6G Edge Intelligence

Federated reinforcement learning: techniques, applications, and open challenges

Toward intelligent resource management in dynamic Fog Computing‐based Internet of Things environment with Deep Reinforcement Learning: A survey

Contact Info

Product

Resources

About