Proxy Experience Replay: Federated Distillation for Distributed Reinforcement Learning

Abstract

Traditional distributed deep reinforcement learning (RL) commonly relies on exchanging the experience replay memory (RM) of each agent. Since the RM contains all state observations and action policy history, it may incur huge communication overhead while violating the privacy of each agent. Alternatively, this article presents a communication-efficient and privacy-preserving distributed RL framework, coined federated reinforcement distillation (FRD). In FRD, each agent exchanges its proxy experience replay memory (ProxRM), in which policies are locally averaged with respect to proxy states clustering actual states. To provide FRD design insights, we present ablation studies on the impact of ProxRM structures, neural network architectures, and communication intervals. Furthermore, we propose an improved version of FRD, coined mixup augmented FRD (MixFRD), in which ProxRM is interpolated using the mixup data augmentation algorithm. Simulations validate the effectiveness of MixFRD in reducing the variance of mission completion time and communication cost, compared to the benchmark schemes, vanilla FRD, federated reinforcement learning (FRL), and policy distillation (PD).

Head Network Distillation: Splitting Distilled Deep Neural Networks for Resource-Constrained Edge Computing Systems

“…Several recent studies investigate techniques to partition DNN models and distribute the processing load. As for distributed computing, Jeong et al [42] and Cha et al [43] discuss distributed training methodologies called federated distillation. Different from these studies, our focus is on split computing for efficient inference rather than training, and our proposed method, head network distillation, can be executed on a single machine.…”

Federated Knowledge Distillation

“…The performance of FRD is evaluated in terms of the mission completion time, and is compared with two baseline distributed RL frameworks: PD [35] and federated reinforcement learning (FRL) that exchanges actor NN model parameters following the standard FL operations [5,6,7,8,36]. Each agent runs an A2C model comprising a pair of actor and critic NNs [37], each of which is a multi-layer perceptron (MLP) with 2 hidden layers.…”

“…Lastly, in that each agent stores a pair of actor and critic NNs, there are three possibilities of exchanging: only actor NNs, critic NNs, or both actor and critic NNs across agents. As seen by several experiments [24,36], exchanging only actor NNs, i.e., policy NNs, achieves the convergence speed as fast as exchanging both actor and critic NNs, while saving the communication cost thanks to ignoring critic NNs. Hereafter we thus focus on an FRD implementation with the experience memory constructed by the actor NN outputs.…”

“…To obviate the aforementioned problems, by leveraging FD, we introduce federated reinforcement distillation (FRD) [24,36], a communication-efficient and privacy-preserving distributed RL framework based on a proxy experience memory. In FRD, each agent stores a local proxy experience memory that consists of a set of pre-arranged proxy states and locally averaged policies.…”

Communication-Efficient Consensus Mechanism for Federated Reinforcement Learning

“…This naive implementation of FL may cause excessive communication overheads between agents and the central server. Recent studies [6], [7] which try to combine distributed DRL with FL mostly focus on improving the involved agents' capability and their collaboration efficiency, while these excessive communication overheads are ignored. In this paper, we introduce the periodic averaging method in FL to alleviate this problem [8], in which agents are allowed to perform several local updates to the model within a period before their local gradients are transmitted to the central server for averaging.…”