FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

Abstract

Federated distillation (FD) is a popular novel algorithmic paradigm for Federated learning (FL), which achieves training performance competitive to prior parameter averaging-based methods, while additionally allowing the clients to train different model architectures, by distilling the client predictions on an unlabeled auxiliary set of data into a student model. In this work, we propose FEDAUX, an extension to FD, which, under the same set of assumptions, drastically improves the performance by deriving maximum utility from the unlabeled auxiliary data. FEDAUX modifies the FD training procedure in two ways: First, unsupervised pre-training on the auxiliary data is performed to find a suitable model initialization for the distributed training. Second, (ε, δ)-differentially private certainty scoring is used to weight the ensemble predictions on the auxiliary data according to the certainty of each client model. Experiments on large-scale convolutional neural networks (CNNs) and transformer models demonstrate that our proposed method achieves remarkable performance improvements over state-ofthe-art FL methods, without adding appreciable computation, communication, or privacy cost. For instance, when training ResNet8 on non-independent identically distributed (i.i.d.) subsets of CIFAR10, FEDAUX raises the maximum achieved validation accuracy from 30.4% to 78.1%, further closing the gap to centralized training performance. Code is available at https://github.com/fedl-repo/fedaux.

One-shot Federated Learning without Server-side Training

“…Currently, most one-shot federated learning methods [9]- [14] rely on Knowledge Distillation [15] or Dataset Distillation [16]. The fundamental idea is to use public data sets or the distilled synthetic data from clients to do model distillation on the server, where the local models play as teachers.…”

“…How to aggregate the knowledge of multiple models into a single model without acquiring their original training datasets is still an open problem. Federated Learning (FL) [4]- [9] was recently proposed as a solution to this challenge and has seen remarkable growth. FL introduces a new machine learning paradigm that allows it to learn from distributed…”

Vertical Federated Principal Component Analysis and Its Kernel Extension on Feature-wise Distributed Data

“…The labels can be expensive and time-consuming to mark, and even require domain expertise [7]. The label can also contain sensitive information, which will incur privacy leakage if not handled properly, especially under the vertically partitioned setting where the labels need to be shared among all clients [8].…”

FedAUXfdp: Differentially Private One-Shot Federated Distillation

“…FedAUX overcomes that challenge, performing remarkably more efficiently on non-iid data than other state-of-the-art federated learning methods, federated averaging, federated proximal learning, Bayesian federated learning, and federated ensemble distillation. For example on MobilenetV2, FedAUX achieves 64.8% server accuracy, while even the second-best method only achieves 46.7% [Sattler et al, 2021a].…”

“…Both governments and private institutions are increasingly interested in securing their data using differential privacy. [Sattler et al, 2021a] train two models on private data, but only privatize the scoring model, leaving the data participating in the classification model exposed. In this work, we add a local, data-level ( , δ)-differentially private mechanism for this second model and give an upper bound on the L 2 -sensitivity of regularized multinomial logistic regression.…”