Communication-Efficient Federated Distillation

Sattler, Felix; Rischke, Roman; Samek, Wojciech

doi:10.48550/arxiv.2012.00632

Cited by 13 publications

(27 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Jeong et. al and subsequent work (Jeong et al, 2018;Itahara et al, 2020;Seo et al, 2020;Sattler et al, 2020a) focus on this aspect. These methods however are computationally more expensive for the resource constrained clients, as distillation needs to be performed locally and perform worse than parameter averaging based training after the same number of communication rounds.…”

Section: A Extended Related Work Discussionmentioning

confidence: 99%

“…Federated Distillation (FD) algorithms, which leverage these distillation techniques to aggregate the client knowledge, are recently gaining popularity, because they outperform conventional parameter averaging based FL methods (Lin et al, 2020;Chen & Chao, 2020) like FEDAVG or FedPROX (McMahan et al, 2017Li et al, 2020a) and allow clients to train heterogeneous model architectures (Li & Wang, 2019;Chang et al, 2019;Li et al, 2021). FD methods can furthermore reduce communication overhead (Jeong et al, 2018;Itahara et al, 2020;Seo et al, 2020;Sattler et al, 2020a), by exploiting the fact that distillation requires only the communication of model predictions instead of full models. In contrast to centralized distillation, where training and distillation data usually coincide, FD makes no restrictions on the auxiliary distillation data 1 , making it widely applicable.…”

Section: Related Workmentioning

confidence: 99%

“…This can be particularly beneficial in situations where clients are running on heterogeneous hardware. Studies show that FD based training has favorable communication properties (Itahara et al, 2020;Sattler et al, 2020a), and can outperform parameter averaging based algorithms (Lin et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

Sattler

Korjakow

Rischke

et al. 2023

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

Federated distillation (FD) is a popular novel algorithmic paradigm for Federated learning (FL), which achieves training performance competitive to prior parameter averaging-based methods, while additionally allowing the clients to train different model architectures, by distilling the client predictions on an unlabeled auxiliary set of data into a student model. In this work, we propose FEDAUX, an extension to FD, which, under the same set of assumptions, drastically improves the performance by deriving maximum utility from the unlabeled auxiliary data. FEDAUX modifies the FD training procedure in two ways: First, unsupervised pre-training on the auxiliary data is performed to find a suitable model initialization for the distributed training. Second, (ε, δ)-differentially private certainty scoring is used to weight the ensemble predictions on the auxiliary data according to the certainty of each client model. Experiments on large-scale convolutional neural networks (CNNs) and transformer models demonstrate that our proposed method achieves remarkable performance improvements over state-ofthe-art FL methods, without adding appreciable computation, communication, or privacy cost. For instance, when training ResNet8 on non-independent identically distributed (i.i.d.) subsets of CIFAR10, FEDAUX raises the maximum achieved validation accuracy from 30.4% to 78.1%, further closing the gap to centralized training performance. Code is available at https://github.com/fedl-repo/fedaux.

show abstract

Section: A Extended Related Work Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

Sattler

Korjakow

Rischke

et al. 2023

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The paper [53] assumes that part of clients have unsupervised data, and trains a convergent model at the server to label them. The paper [54] considers using shared unlabeled data for Federated Distillation [55,56]. Another related work [57] trains and aggregates the model parameters of the labeled server, and unlabeled clients in parallel.…”

Section: Related Workmentioning

confidence: 99%

SemiFL: Communication Efficient Semi-Supervised Federated Learning with Unlabeled Clients

Diao¹,

Ding²,

Tarokh³

2021

Preprint

View full text Add to dashboard Cite

Federated Learning allows training machine learning models by using the computation and private data resources of a large number of distributed clients such as smartphones and IoT devices. Most existing works on Federated Learning (FL) assume the clients have ground-truth labels. However, in many practical scenarios, clients may be unable to label task-specific data, e.g., due to lack of expertise. In this work, we consider a server that hosts a labeled dataset, and wishes to leverage clients with unlabeled data for supervised learning. We propose a new Federated Learning framework referred to as SemiFL in order to address the problem of Semi-Supervised Federated Learning (SSFL). In SemiFL, clients have completely unlabeled data, while the server has a small amount of labeled data. SemiFL is communication efficient since it separates the training of server-side supervised data and client-side unsupervised data. We demonstrate various efficient strategies of SemiFL that enhance learning performance. Extensive empirical evaluations demonstrate that our communication efficient method can significantly improve the performance of a labeled server with unlabeled clients. Moreover, we demonstrate that SemiFL can outperform many existing FL results trained with fully supervised data, and perform competitively with the state-of-the-art centralized Semi-Supervised Learning (SSL) methods. For instance, in standard communication efficient scenarios, our method can perform 93% accuracy on the CIFAR10 dataset with only 4000 labeled samples at the server. Such accuracy is only 2% away from the result trained from 50000 fully labeled data, and it improves about 30% upon existing SSFL methods in the communication efficient setting.Preprint. Under review.

show abstract

“…The scholars then focus to perform the model compression while still following the federated learning paradigm. The knowledge distillation then has been applied to transfer local model information while keeping the model size small at the local devices in [48]. It utilises knowledge distillation instead of model parameter fusion to update parameters.…”

Section: Classificationmentioning

confidence: 99%

Federated Learning Meets Natural Language Processing: A Survey

Liu,

Ho,

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated Learning aims to learn machine learning models from multiple decentralized edge devices (e.g. mobiles) or servers without sacrificing local data privacy. Recent Natural Language Processing techniques rely on deep learning and large pre-trained language models. However, both big deep neural and language models are trained with huge amounts of data which often lies on the server side. Since text data is widely originated from end users, in this work, we look into recent NLP models and techniques which use federated learning as the learning framework. Our survey discusses major challenges in federated natural language processing, including the algorithm challenges, system challenges as well as the privacy issues. We also provide a critical review of the existing Federated NLP evaluation methods and tools. Finally, we highlight the current research gaps and future directions.

show abstract

Communication-Efficient Federated Distillation

Cited by 13 publications

References 33 publications

FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

SemiFL: Communication Efficient Semi-Supervised Federated Learning with Unlabeled Clients

Federated Learning Meets Natural Language Processing: A Survey

Contact Info

Product

Resources

About