Asynchronous Upper Confidence Bound Algorithms for Federated Linear Bandits

Abstract

Linear contextual bandit is a popular online learning problem. It has been mostly studied in centralized learning settings. With the surging demand of large-scale decentralized model learning, e.g., federated learning, how to retain regret minimization while reducing communication cost becomes an open challenge. In this paper, we study linear contextual bandit in a federated learning setting. We propose a general framework with asynchronous model update and communication for a collection of homogeneous clients and heterogeneous clients, respectively. Rigorous theoretical analysis is provided about the regret and communication cost under this distributed learning framework; and extensive empirical evaluations demonstrate the effectiveness of our solution.

Federated Contextual Cascading Bandits with Asynchronous Communication and Heterogeneous Users

“…The key challenge of the algorithm design is to manage the data inconsistency during user clustering. Existing matrix determinant-based protocols (Li and Wang 2021;Liu et al 2022a;He et al 2022) fail to achieve this goal due to insufficient communication, so we propose a novel communication protocol that employs a p t -auxiliary protocol in conjunction with the matrix determinant-based protocol. This new protocol effectively controls data inconsistency, ensuring the correct operation of heterogeneity testing and action selection.…”

Communication Efficient Federated Learning for Generalized Linear Bandits

“…GLB under federated/distributed setting still remains under-explored. The most related works in bandit literature are the federated/distributed linear bandits (Korda et al, 2016;Dubey and Pentland, 2020;Huang et al, 2021;Li and Wang, 2021). In these works, thanks to the existence of closed-form solution for linear models, the clients only communicate their local sufficient statistics for global model update.…”

“…Huang et al (2021) considered a star-shaped communication network as in our paper, but they assumed a fixed arm set setting and thus proposed a phase-based elimination algorithm. The closest works to ours are Dubey and Pentland, 2020;Li and Wang, 2021), which proposed event-triggered communication protocols to obtain sub-linear communication cost over time for federated linear bandit with a time-varying arm set.…”

“…As a classic model for sequential decision making problems, contextual bandit has been widely used for a variety of real-world applications, including recommender systems (Li et al, 2010a), display advertisement (Li et al, 2010b) and clinical trials (Durand et al, 2018). While most existing bandit solutions are designed under a centralized setting (i.e., data is readily available at a central server), in response to the increasing application scale and public concerns of privacy, there is increasing research effort on federated bandit learning lately Dubey and Pentland, 2020;Shi et al, 2021;Huang et al, 2021;Li and Wang, 2021), where N clients collaborate with limited communication bandwidth to minimize the overall cumulative regret incurred over a finite time horizon T , while keeping each client's raw data local. Compared with standard federated learning (McMahan et al, 2017;Kairouz et al, 2019) that works with fixed datasets, federated bandit learning is characterized by its online interactions with the environment, which continuously provides new data samples to the clients over time.…”

“…Existing federated bandit learning solutions only partially addressed this challenge by considering simple bandit models, like context-free bandit (Shi et al, 2021) and contextual linear bandit Dubey and Pentland, 2020;Li and Wang, 2021), where closed-form solution for both local and global model update exists. Therefore, efficient communication for global bandit model update is realized by directly aggregating local sufficient statistics, such that the only concern left is how to control the communication frequency over time horizon T .…”