Federated reinforcement learning: techniques, applications, and open challenges

Abstract

This paper presents a comprehensive survey of Federated Reinforcement Learning (FRL), an emerging and promising field in Reinforcement Learning (RL). Starting with a tutorial of Federated Learning (FL) and RL, we then focus on the introduction of FRL as a new method with great potential by leveraging the basic idea of FL to improve the performance of RL while preserving data-privacy. According to the distribution characteristics of the agents in the framework, FRL algorithms can be divided into two categories, i.e., Horizontal Federated Reinforcement Learning (HFRL) and Vertical Federated Reinforcement Learning (VFRL). We provide the detailed definitions of each category by formulas, investigate the evolution of FRL from a technical perspective, and highlight its advantages over previous RL algorithms. In addition, the existing works on FRL are summarized by application fields, including edge computing, communication, control optimization, and attack detection. Finally, we describe and discuss several key research directions that are crucial to solving the open problems within FRL.

FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergence

“…The TV penalty occurred when investigating the connection between the local and global policy advantage, i.e., Theorem I, to tackle the data heterogeneity, while the KL penalty came from (12) to constrain the policy update at one agent.…”

“…Due to the distributed data and computing power in large-scale applications such as autonomous driving, the training of RL algorithms under the FL framework is inevitable. Unfortunately, many challenges faced by supervised FL, e.g., the data heterogeneity and communication bottleneck, are still valid and even worse for FRL [12]. For example, centralized policy gradient methods already suffer from high variance which is detrimental to the convergence speed and training performance [13]- [15], and data heterogeneity imposes another layer of difficulty for the convergence of FRL.…”

“…The heterogeneity issue of FRL is different from that of the supervised FL and the related study is very limited [21]- [23]. There are mainly two types of RL methods, namely, the action-value method and the policy gradient (PG) method [11], [12]. The first performs action selection according to the value function, while the second utilizes ML models to approximate the policy and update the policy by gradient based algorithms.…”

AVDDPG – Federated reinforcement learning applied to autonomous platoon control

“…HFRL may be noted to have similarities to "Parallel RL". Parallel RL is a long studied field of RL, where agent gradients are transferred amongst each other [5,10,11] .…”

Just Round: Quantized Observation Spaces Enable Memory Efficient Learning of Dynamic Locomotion

“…Still, many robots, especially those involved in tasks as consequential as search-and-rescue and space exploration [8], will have to adapt to ever-changing environmental conditions and continue to optimize and update their internal policies over the course of their lifetime [9]. As such, to untether these methods from the confines of a lab, data collection and storage must be memory efficient enough to allow for either low-latency networking [10], [11] or for sufficient experience data to be stored on-board edge computing devices [12]. As fast and secure updates may not be possible in remote locations or when using bandwidth-constrained or highlatency cloud networks [13], it is imperative to find ways to reduce the overall memory footprint of DRL training.…”