Federated Knowledge Distillation

Seo, Hyowoon; Park, Jihong; Oh, Seungeun; Bennis, Mehdi; Kim, Seong-Lyun

doi:10.48550/arxiv.2011.02367

Cited by 21 publications

(24 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Jeong et. al and subsequent work (Jeong et al, 2018;Itahara et al, 2020;Seo et al, 2020;Sattler et al, 2020a) focus on this aspect. These methods however are computationally more expensive for the resource constrained clients, as distillation needs to be performed locally and perform worse than parameter averaging based training after the same number of communication rounds.…”

Section: A Extended Related Work Discussionmentioning

confidence: 99%

“…Federated Distillation (FD) algorithms, which leverage these distillation techniques to aggregate the client knowledge, are recently gaining popularity, because they outperform conventional parameter averaging based FL methods (Lin et al, 2020;Chen & Chao, 2020) like FEDAVG or FedPROX (McMahan et al, 2017Li et al, 2020a) and allow clients to train heterogeneous model architectures (Li & Wang, 2019;Chang et al, 2019;Li et al, 2021). FD methods can furthermore reduce communication overhead (Jeong et al, 2018;Itahara et al, 2020;Seo et al, 2020;Sattler et al, 2020a), by exploiting the fact that distillation requires only the communication of model predictions instead of full models. In contrast to centralized distillation, where training and distillation data usually coincide, FD makes no restrictions on the auxiliary distillation data 1 , making it widely applicable.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

Sattler

Korjakow

Rischke

et al. 2023

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Federated distillation (FD) is a popular novel algorithmic paradigm for Federated learning (FL), which achieves training performance competitive to prior parameter averaging-based methods, while additionally allowing the clients to train different model architectures, by distilling the client predictions on an unlabeled auxiliary set of data into a student model. In this work, we propose FEDAUX, an extension to FD, which, under the same set of assumptions, drastically improves the performance by deriving maximum utility from the unlabeled auxiliary data. FEDAUX modifies the FD training procedure in two ways: First, unsupervised pre-training on the auxiliary data is performed to find a suitable model initialization for the distributed training. Second, (ε, δ)-differentially private certainty scoring is used to weight the ensemble predictions on the auxiliary data according to the certainty of each client model. Experiments on large-scale convolutional neural networks (CNNs) and transformer models demonstrate that our proposed method achieves remarkable performance improvements over state-ofthe-art FL methods, without adding appreciable computation, communication, or privacy cost. For instance, when training ResNet8 on non-independent identically distributed (i.i.d.) subsets of CIFAR10, FEDAUX raises the maximum achieved validation accuracy from 30.4% to 78.1%, further closing the gap to centralized training performance. Code is available at https://github.com/fedl-repo/fedaux.

show abstract

Section: A Extended Related Work Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

Sattler

Korjakow

Rischke

et al. 2023

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…The output of each model is regularized to be similar to the ensemble of predictions from all models via a distillation loss. The idea of codistillation is used by several methods to reduce communication cost of federated learning (Sui et al, 2020;Li and Wang, 2019;Seo et al, 2020; Lin et al, 2020b;Sun and Lyu, 2020). For example, Sui et al (2020) proposed a federated ensemble distillation approach for medical relation extraction.…”

Section: Communication Efficient Flmentioning

confidence: 99%

FedKD: Communication Efficient Federated Learning via Knowledge Distillation

Wu,

Lyu

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning is widely used to learn intelligent models from decentralized data. In federated learning, clients need to communicate their local model updates in each iteration of model learning. However, model updates are large in size if the model contains numerous parameters, and there usually needs many rounds of communication until model converges. Thus, the communication cost in federated learning can be quite heavy. In this paper, we propose a communication efficient federated learning method based on knowledge distillation. Instead of directly communicating the large models between clients and server, we propose an adaptive mutual distillation framework to reciprocally learn a student and a teacher model on each client, where only the student model is shared by different clients and updated collaboratively to reduce the communication cost. Both the teacher and student on each client are learned on its local data and the knowledge distilled from each other, where their distillation intensities are controlled by their prediction quality. To further reduce the communication cost, we propose a dynamic gradient approximation method based on singular value decomposition to approximate the exchanged gradients with dynamic precision. Extensive experiments on benchmark datasets in different tasks show that our approach can effectively reduce the communication cost and achieve competitive results.

show abstract

“…The NTK framework has been mostly used for convergence analyses in FL. Seo et al (2020) studied two knowledge distillation methods in FL and compared their convergence properties based on the neural network function evolution in the NTK regime. incorporated batch normalization layers to local models, and provided theoretical justification for its faster convergence by studying the minimum nonnegative eigenvalue of the tangent kernel matrix.…”

Section: Related Workmentioning

confidence: 99%

Neural Tangent Kernel Empowered Federated Learning

Yue¹,

Jin²,

Pilgrim³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning (FL) is a privacy-preserving paradigm where multiple participants jointly solve a machine learning problem without sharing raw data. Unlike traditional distributed learning, a unique characteristic of FL is statistical heterogeneity, namely, data distributions across participants are different from each other. Meanwhile, recent advances in the interpretation of neural networks have seen a wide use of neural tangent kernel (NTK) for convergence and generalization analyses. In this paper, we propose a novel FL paradigm empowered by the NTK framework. The proposed paradigm addresses the challenge of statistical heterogeneity by transmitting update data that are more expressive than those of the traditional FL paradigms. Specifically, sample-wise Jacobian matrices, rather than model weights/gradients, are uploaded by participants. The server then constructs an empirical kernel matrix to update a global model without explicitly performing gradient descent. We further develop a variant with improved communication efficiency and enhanced privacy. Numerical results show that the proposed paradigm can achieve the same accuracy while reducing the number of communication rounds by an order of magnitude compared to federated averaging.

show abstract

Federated Knowledge Distillation

Cited by 21 publications

References 26 publications

FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

FedKD: Communication Efficient Federated Learning via Knowledge Distillation

Neural Tangent Kernel Empowered Federated Learning

Contact Info

Product

Resources

About