Leveraging Communication Topologies Between Learning Agents in Deep Reinforcement Learning

Abstract

A common technique to improve speed and robustness of learning in deep reinforcement learning (DRL) and many other machine learning algorithms is to run multiple learning agents in parallel. A neglected component in the development of these algorithms has been how best to arrange the learning agents involved to better facilitate distributed search. Here we draw upon results from the networked optimization and collective intelligence literatures suggesting that arranging learning agents in less than fully connected topologies (the implicit way agents are commonly arranged in) can improve learning. We explore the relative performance of four popular families of graphs and observe that one such family (Erdos-Renyi random graphs) empirically outperforms the standard fully-connected communication topology across several DRL benchmark tasks. We observe that 1000 learning agents arranged in an Erdos-Renyi graph can perform as well as 3000 agents arranged in the standard fully-connected topology, showing the large learning improvement possible when carefully designing the topology over which agents communicate. We complement these empirical results with a preliminary theoretical investigation of why less than fully connected topologies can perform better. Overall, our work suggests that distributed machine learning algorithms could be made more efficient if the communication topology between learning agents was optimized.We focus on communication topology because it has been shown to result in increased exploration, higher overall maximum reward, and higher diversity of solutions in both simulated high-dimensional optimization problems (Lazer & Friedman, 2007) and human experiments (Barkoczi & Galesic, 2016), and because, to the best of our knowledge, almost no prior work has investigated how the topology of communication between agents affects learning performance in distributed Deep Reinforcement Learning (DRL). The two topologies that are almost always used are either a complete (fully-connected) network, in which all processors communicate with each other; or a star network-in which all processors communicate with a single hub server, which is, in effect, a more efficient, centralized implemen-

TAPE: Leveraging Agent Topology for Cooperative Multi-Agent Policy Gradient

“…We now establish policy improvement theorem of stochastic TAPE, and prove a theorem for the cooperation improvement from the perspective of exploring the parameter space, which is a common motivation in RL research (Schulman, Chen, and Abbeel 2017;Haarnoja et al 2018;Zhang et al 2021;Adjodah et al 2019). We assume the policy to have tabular expressions.…”

“…As the global Q value is determined by the centralized critic for all agents, sub-optimal actions of one agent will easily influence all others. Topology in Reinforcement Learning Adjodah et al (Adjodah et al 2019) discuss the communication topology issue in parallel-running RL algorithms such as A3C (Mnih et al 2016). Results show that the centralized learner implicitly yields a fully-connected communication topology among parallel workers, which will harm their performance.…”

Communication-Efficient Edge AI: Algorithms and Systems

“…• Fast aggregation via over-the-air computation [21], [84], [85], [86] • Aggregation frequency control with limited bandwidth and computation resources [87], [88], [89] • Data reshuffling via index coding and pliable index coding for improving training performance [90], [91], [92] • Straggler mitigation via coded computing [93], [94], [95], [96], [97], [98], [99], [100], [101] • Training in decentralized system mode [102], [103], [104], [105], [106], [107], [108], [109], [110], [111], [112] Model Partition Based Edge Training Systems…”

“…There have been several works demonstrating that some carefully designed topologies of networks achieve better performance than the fully connected network. It has been empirically observed in [107] that using an alternative network topology between devices can lead to improved learning performance in several deep reinforcement learning tasks compared with the standard fully-connected communication topology. Specifically, it was observed in [107] that the Erdos-Renyi graph topology with 1000 devices can compete with the standard fully-connected topology with 3000 devices, which shows that the machine learning performance can be more efficient if the topology is carefully designed.…”

“…It has been empirically observed in [107] that using an alternative network topology between devices can lead to improved learning performance in several deep reinforcement learning tasks compared with the standard fully-connected communication topology. Specifically, it was observed in [107] that the Erdos-Renyi graph topology with 1000 devices can compete with the standard fully-connected topology with 3000 devices, which shows that the machine learning performance can be more efficient if the topology is carefully designed. Considering that different devices may require different times to carry out local computation, Neglia et al [108] analyzed the influences of different network topologies on the total runtime of distributed subgradient methods, which can determine the degrees of the topology graph, leading to the faster convergence speed.…”