Federated Multi-Armed Bandits

Abstract

Federated multi-armed bandits (FMAB) is a new bandit paradigm that parallels the federated learning (FL) framework in supervised learning. It is inspired by practical applications in cognitive radio and recommender systems, and enjoys features that are analogous to FL. This paper proposes a general framework of FMAB and then studies two specific federated bandit models. We first study the approximate model where the heterogeneous local models are random realizations of the global model from an unknown distribution. This model introduces a new uncertainty of client sampling, as the global model may not be reliably learned even if the finite local models are perfectly known. Furthermore, this uncertainty cannot be quantified a priori without knowledge of the suboptimality gap. We solve the approximate model by proposing Federated Double UCB (Fed2-UCB), which constructs a novel “double UCB” principle accounting for uncertainties from both arm and client sampling. We show that gradually admitting new clients is critical in achieving an O(log(T)) regret while explicitly considering the communication loss. The exact model, where the global bandit model is the exact average of heterogeneous local models, is then studied as a special case. We show that, somewhat surprisingly, the order-optimal regret can be achieved independent of the number of clients with a careful choice of the update periodicity. Experiments using both synthetic and real-world datasets corroborate the theoretical analysis and demonstrate the effectiveness and efficiency of the proposed algorithms.

Pervasive AI for IoT Applications: A Survey on Resource-Efficient Distributed Artificial Intelligence

“…Since most of the MAB algorithms discussed in this paper are recent [143], [144], [147], [149], it remains interesting to see their implications on practical applications, for example, quantifying the effect of bounded communication resources or energy used in wearable devices and congestion between edge nodes. Similarly, quantifying the improvement in regret bounds on actual and Quality of Experience (QoE) metrics can be promising.…”

“…Furthermore, privacy concerns sometimes limit the ability of these local servers to share data with other servers. The work in [149] studies the case of a set of servers that run a recommender system for their prospective clients. The goal of each one is to recommend the most popular content across all servers.…”

“…• Towards the federation of the bandit framework: When the bandit instances faced by each agent are local biased instances, the federated bandits framework arises. In such a situation, agents need to learn with the help of a logically centralized controller, similar to supervised federated learning, in order to estimate the true global instance and the true best action [149]. However, if agents are not interested in solving a hidden global instance but rather only their own, they may reuse their peers' experience and an instance-similarity metric to help them solve their own instances [150].…”

Accelerating Federated Learning With a Global Biased Optimiser

“…Therefore, there has been much research interest in developing new FL algorithms that exhibit superior convergence (in terms of communication rounds) compared to FedAvg in the face of non-IID client data. Although most works consider training Deep [2] Averages client models x x O(|x|) FedProx [6] Proximal term for local objectives x x O(2|x|) FedMAX [7] Max-entropy term for local objectives x x O(|x|) AdaptiveFedOpt [3] Server-only optimiser x x O(|x|) MFL [8] Averages client models and optimisers x, s x, s O(|x| + |s|) Mimelite [9] Unbiased global optimiser Neural Networks (DNNs) in a round-based synchronous fashion, some works propose asynchronous algorithms to reduce training time [10], [11], and for other models such as Random Forests [12] and Multi-Armed Bandits [13].…”

Multi-Armed Bandit for Link Configuration in Millimeter-Wave Networks: An Approach for Solving Sequential Decision-Making Problems

“…(b) The evaluation result in scenario III. 12 ||| IEEE VEHICULAR TECHNOLOGY MAGAZINE | MONTH 2023algorithm[15], which is inspired by the principle of federated learning: training a model across multiple decentralized agents and merging the locally trained models in a communication-efficient and data-private way. Another related challenge to a federated MAB is the adversarial users who could attack the systems by sharing malicious local models, which calls for designing defense mechanisms to protect the global model.…”