FedGH: Heterogeneous Federated Learning with Generalized Global Header

Abstract

Federated learning (FL) is an emerging machine learning paradigm that allows multiple parties to train a shared model collaboratively in a privacy-preserving manner. Existing horizontal FL methods generally assume that the FL server and clients hold the same model structure. However, due to system heterogeneity and the need for personalization, enabling clients to hold models with diverse structures has become an important direction. Existing model-heterogeneous FL approaches often require publicly available datasets and incur high communication and/or computational costs, which limit their performances. To address these limitations, we propose the Federated Global prediction Header (FedGH) approach. It is a communication and computation-efficient modelheterogeneous FL framework which trains a shared generalized global prediction header with representations extracted by heterogeneous extractors for clients' models at the FL server. The trained generalized global prediction header learns from different clients. The acquired global knowledge is then transferred to clients to substitute each client's local prediction header. We derive the nonconvex convergence rate of FedGH. Extensive experiments on two real-world datasets demonstrate that FedGH achieve significantly more advantageous performance in both model-homogeneous and -heterogeneous FL scenarios compared to seven state-of-the-art personalized FL models, beating the best-performing baseline by up to 8.87% (for model-homogeneous FL) and 1.83% (for modelheterogeneous FL) in terms of average test accuracy, while saving up to 85.53% of communication overhead.

FedBEVT: Federated Learning Bird's Eye View Perception Transformer in Road Traffic Systems

“…This method decouples the model's parameters, allowing only a subset of parameters to be shared and aggregated among clients, while the private parameters are learned solely on local data. Previous research has attempted to select these private parameters based on model architecture [41]- [44] or data similarities [45]- [47]. However, none of these methods have been proposed specifically for training personalized transformer models, which have a significantly different structure compared to other machine learning models.…”

FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning

“…However, existing prototype-based HtFL methods naively aggregate heterogeneous client prototypes on the server using weighted-averaging, which has several limitations. First, the weighted-averaging protocol requires clients to upload class distribution information of private data to the server as weights, which leaks sensitive distribution information about clients' data (Yi et al 2023). Secondly, the prototypes generated from heterogeneous clients have diverse scales and separation margins.…”