Delay-Optimal Traffic Engineering through Multi-agent Reinforcement Learning

Pinyoanuntapong, Pinyarash; Lee, Minwoo; Wang, Pu

doi:10.1109/infcomw.2019.8845154

Cited by 27 publications

(6 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To solve the above MA-MDP problem, we exploit the multi-agent reinforcement learning, where the routers (agents) distributively learn the optimal target forwarding policy 𝜋 𝜋 𝜋 to minimize the average server-worker delay. To implement the multi-agent reinforcement learning algorithm, we adopt a distributed actor-critic architecture similar to asynchronous advantage actor-critic (A3C) [26], [27], where each router individually runs a local critic and a local actor,…”

Section: B Convergence Optimization Via Multi-agent Reinforcement Lea...mentioning

confidence: 99%

“…Model and Dataset: We use FEMNIST, the federated version of MNIST [41] on the LEAF [30] character recognition task, where LEAF is a benchmarking framework for federated learning. FEMNIST consists of handwritten digits (10), uppercase (26), and lowercase (26) letters leading to a total of 62 classes with each image having 28×28 pixels. The whole dataset is partitioned into 3550 data portions/users with Non-IID data distribution.…”

Section: A Experiments Setupmentioning

confidence: 99%

See 1 more Smart Citation

EdgeML: Towards Network-Accelerated Federated Learning over Wireless Edge

Pinyoanuntapong¹,

Jayagopal²,

Balakrishnan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning (FL) is a distributed machine learning technology for next-generation AI systems that allows a number of workers, i.e., edge devices, collaboratively learn a shared global model while keeping their data locally to prevent privacy leakage. Enabling FL over wireless multi-hop networks can democratize AI and make it accessible in a costeffective manner. However, the noisy bandwidth-limited multihop wireless connections can lead to delayed and nomadic model updates, which significantly slows down the FL convergence speed. To address such challenges, this paper aims to accelerate FL convergence over wireless edge by optimizing the multi-hop federated networking performance. In particular, the FL convergence optimization problem is formulated as a Markov decision process (MDP). To solve such MDP, multiagent reinforcement learning (MA-RL) algorithms along with domain-specific action space refining schemes are developed, which online learn the delay-minimum forwarding paths to minimize the model exchange latency between the edge devices (i.e., workers) and the remote server. To validate the proposed solutions, FedEdge is developed and implemented, which is the first experimental framework in the literature for FL over multihop wireless edge computing networks. FedEdge allows us to fast prototype, deploy, and evaluate novel FL algorithms along with RL-based system optimization methods in real wireless devices. Moreover, a physical experimental testbed is implemented by customizing the widely adopted Linux wireless routers and ML computing nodes. Such testbed can provide valuable insights into the practical performance of FL in the field. Finally, our experimentation results on the testbed show that the proposed networkaccelerated FL system can practically and significantly improve FL convergence speed, compared to the FL system empowered by the production-grade commercially-available wireless networking protocol, BATMAN-Adv. I. INTRODUCTION:Distributed machine learning, specifically federated learning (FL), has been envisioned as a key technology for enabling next-generation AI at scale. FL significantly reduces privacy risks and communication costs, which are critical in modern AI systems. FL allows workers (i.e., edge devices) to collaboratively learn a global model and maintains the locality

show abstract

Section: B Convergence Optimization Via Multi-agent Reinforcement Lea...mentioning

confidence: 99%

Section: A Experiments Setupmentioning

confidence: 99%

EdgeML: Towards Network-Accelerated Federated Learning over Wireless Edge

Pinyoanuntapong¹,

Jayagopal²,

Balakrishnan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The deep Q-routing algorithm proposed in [148] uses the regular TD prediction to calculate the expected long term return, which may lead to high bias since the TD prediction only considers the impact of the next-hop on the expected return. To overcome this issue, the authors in [150] propose a spatial difference (SD) prediction approach for the packet routing problem. The SD prediction approach leverages the number of hops from the current router node as the standard to outline different Q-value estimation methods, e.g., 1-hop or nhop action-value estimation.…”

Section: Packet Routingmentioning

confidence: 99%

Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey

Li¹,

Zhu²,

Luong³

et al. 2021

Preprint

View full text Add to dashboard Cite

Future Internet involves several emerging technologies such as 5G and beyond 5G networks, vehicular networks, unmanned aerial vehicle (UAV) networks, and Internet of Things (IoTs). Moreover, future Internet becomes heterogeneous and decentralized with a large number of involved network entities. Each entity may need to make its local decision to improve the network performance under dynamic and uncertain network environments. Standard learning algorithms such as single-agent Reinforcement Learning (RL) or Deep Reinforcement Learning (DRL) have been recently used to enable each network entity as an agent to learn an optimal decision-making policy adaptively through interacting with the unknown environments. However, such an algorithm fails to model the cooperations or competitions among network entities, and simply treats other entities as a part of the environment that may result in the non-stationarity issue. Multi-agent Reinforcement Learning (MARL) allows each network entity to learn its optimal policy by observing not only the environments, but also other entities' policies. As a result, MARL can significantly improve the learning efficiency of the network entities, and it has been recently used to solve various issues in the emerging networks. In this paper, we thus review the applications of MARL in the emerging networks. In particular, we provide a tutorial of MARL and a comprehensive survey of applications of MARL in next generation Internet. In particular, we first introduce single-agent RL and MARL. Then, we review a number of applications of MARL to solve emerging issues in future Internet. The issues consist of network access, transmit power control, computation offloading, content caching, packet routing, trajectory design for UAV-aided networks, and network security issues. Finally, we discuss the challenges, open issues, and future directions related to the applications of MARL in future Internet.

show abstract

“…Precisely, MDP represents and abstract representation of learning problems through interaction to achieve a target control and optimization goal. This work addresses the objective by representing the TE problem as a multi-agent MDP (MA-MDP) [157][158][159].…”

Section: Rnn-lstm Inference Model Creationmentioning

confidence: 99%

AI-Assisted Framework for Green-Routing and Load Balancing in Hybrid Software-Defined Networking: Proposal, Challenges and Future Perspective

et al. 2020

View full text Add to dashboard Cite

The explosive growth of IP networks, the advent of cloud computing, and the rapid progress in wireless communications witnessed today reflect significant progress towards meeting the explosive data traffic demands. Consequently, communications service providers should deploy efficient and intelligent network solutions to accommodate the huge traffic demands and to ease the capacity pressure on their network infrastructure. Besides, vendors should develop novel energy-efficient networks to reduce network utility costs and carbon footprint. Software-defined networking (SDN) provides a suitable solution, however, complete SDN deployment is currently unachievable in the short-term. An alternative is the hybrid SDN/ open shortest path forwarding (OSPF) network, which allows the deployment of SDN in legacy networks. Nevertheless, hybrid SDN/OSPF also faces several technical, economic and organizational challenges. Although many energyefficiency routing solutions exist in hybrid SDN/OSPF networks, they are generic and reactive by design. Moreover, these solutions are characterized by manual control plane forwarding configurations, leading to sub-optimal performance. The recent promising combination of SDN and artificial intelligence (AI) techniques such as machine learning (ML) and deep learning (DL) in traffic management and control provides tremendous opportunities. In this paper, we first provide a review of the most recent optimization approaches for global energy-efficient routing and load balancing. Next, we investigate a scalable and intelligent integrated architectural framework that leverages deep reinforcement learning (DRL) techniques to realize predictive and rate adaptive energy-efficient routing with guaranteed quality of service (QoS), in transitional hybrid SDN/OSPF networks. Based on the need to minimize global network energy consumption and improve link performance, this paper provides key research insights into the current progress in hybrid SDN/OSPF, ML and AI in the hope of stimulating more research.

show abstract

Delay-Optimal Traffic Engineering through Multi-agent Reinforcement Learning

Cited by 27 publications

References 18 publications

EdgeML: Towards Network-Accelerated Federated Learning over Wireless Edge

EdgeML: Towards Network-Accelerated Federated Learning over Wireless Edge

Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey

AI-Assisted Framework for Green-Routing and Load Balancing in Hybrid Software-Defined Networking: Proposal, Challenges and Future Perspective

Contact Info

Product

Resources

About