Ternary Compression for Communication-Efficient Federated Learning

Xu, Jinjin; Du, Wenli; Jin, Yaochu; He, Wangli; Cheng, Ran

doi:10.1109/tnnls.2020.3041185

Cited by 128 publications

(69 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For large-scale models like the ones described above, the communication overhead of running the Federated Averaging algorithm can become a prohibitive bottleneck. Although a wide variety of methods to reduce the communication overhead in Federated Averaging have been proposed, including approaches that reduce the frequency of communication [1], use client sampling [1], [21], neural network pruning [22], message sparsification [23], [24], [25] and other lossy [26], [27], [28], [24], [29] and lossless compression techniques [30], [31], the fundamental issue of scaling to larger models persists.…”

Section: A Federated Averagingmentioning

confidence: 99%

See 1 more Smart Citation

CFD: Communication-Efficient Federated Distillation via Soft-Label Quantization and Delta Coding

Sattler

Rischke

Samek

2022

IEEE Trans. Netw. Sci. Eng.

View full text Add to dashboard Cite

Communication constraints are one of the major challenges preventing the wide-spread adoption of Federated Learning systems. Recently, Federated Distillation (FD), a new algorithmic paradigm for Federated Learning with fundamentally different communication properties, emerged. FD methods leverage ensemble distillation techniques and exchange model outputs, presented as soft labels on an unlabeled public data set, between the central server and the participating clients. In this work, we investigate FD from the perspective of communication efficiency by analyzing the effects of active distillation-data curation, soft-label quantization, and delta-coding techniques. Based on the insights gathered from this analysis, we present Compressed Federated Distillation (CFD), an efficient Federated Distillation method. Extensive experiments, on Federated image classification and language modeling problems, at different levels of data heterogeneity, demonstrate that our method can reduce the amount of communication necessary to achieve fixed performance targets by more than two orders of magnitude when compared to FD, and by more than four orders of magnitude when compared to parameter averaging based techniques like Federated Averaging.

show abstract

Section: A Federated Averagingmentioning

confidence: 99%

“…Quantization is a popular technique to reduce communication and has been successfully applied in Federated Averaging to reduce the size of the parameter updates [14], [24], [29]. Quantization techniques, however, so far have not been applied to Federated Distillation.…”

Section: B Soft-label Quantizationmentioning

confidence: 99%

CFD: Communication-Efficient Federated Distillation via Soft-Label Quantization and Delta Coding

Sattler

Rischke

Samek

2022

IEEE Trans. Netw. Sci. Eng.

View full text Add to dashboard Cite

show abstract

“…Federated learning [33] originates from the attempt to address the privacy concerns in distributed learning [30], which has been widely studied and applied in real-world applications due to its capability of privacy protection and parallel computing [50,60]. Generally, the global optimization objective of a federated learning system can be written by min…”

Section: Federated Learningmentioning

confidence: 99%

A federated data-driven evolutionary algorithm for expensive multi-/many-objective optimization

Jin

2021

Complex Intell. Syst.

Self Cite

View full text Add to dashboard Cite

Data-driven optimization has found many successful applications in the real world and received increased attention in the field of evolutionary optimization. Most existing algorithms assume that the data used for optimization are always available on a central server for construction of surrogates. This assumption, however, may fail to hold when the data must be collected in a distributed way and are subject to privacy restrictions. This paper aims to propose a federated data-driven evolutionary multi-/many-objective optimization algorithm. To this end, we leverage federated learning for surrogate construction so that multiple clients collaboratively train a radial-basis-function-network as the global surrogate. Then a new federated acquisition function is proposed for the central server to approximate the objective values using the global surrogate and estimate the uncertainty level of the approximated objective values based on the local models. The performance of the proposed algorithm is verified on a series of multi-/many-objective benchmark problems by comparing it with two state-of-the-art surrogate-assisted multi-objective evolutionary algorithms.

show abstract

“…A number of prior studies in [3][4][5][6][7][8][9][10][11][12][13][14][15][16] have investigated important problems related to wireless network optimization of FL. The works in [3][4][5][6] provided a comprehensive survey of existing studies and summarized open problems in FL.…”

Section: Related Workmentioning

confidence: 99%

“…One key challenge is the contradiction between the huge communication costs required by FL parameter transmission and the limited available communication resources [5]. Therefore, on one hand, the existing studies in [7][8][9][10][11][12] proposed to compress the FL model parameters to reduce the communication cost. In particular, the authors of [7] proposed a sparsification and quantization method that compresses the trained FL model.…”

Section: Related Workmentioning

confidence: 99%

Wireless Network Optimization for Federated Learning with Model Compression in Hybrid VLC/RF Systems

Huang

Yang

Chen

et al. 2021

Entropy

View full text Add to dashboard Cite

In this paper, the optimization of network performance to support the deployment of federated learning (FL) is investigated. In particular, in the considered model, each user owns a machine learning (ML) model by training through its own dataset, and then transmits its ML parameters to a base station (BS) which aggregates the ML parameters to obtain a global ML model and transmits it to each user. Due to limited radio frequency (RF) resources, the number of users that participate in FL is restricted. Meanwhile, each user uploading and downloading the FL parameters may increase communication costs thus reducing the number of participating users. To this end, we propose to introduce visible light communication (VLC) as a supplement to RF and use compression methods to reduce the resources needed to transmit FL parameters over wireless links so as to further improve the communication efficiency and simultaneously optimize wireless network through user selection and resource allocation. This user selection and bandwidth allocation problem is formulated as an optimization problem whose goal is to minimize the training loss of FL. We first use a model compression method to reduce the size of FL model parameters that are transmitted over wireless links. Then, the optimization problem is separated into two subproblems. The first subproblem is a user selection problem with a given bandwidth allocation, which is solved by a traversal algorithm. The second subproblem is a bandwidth allocation problem with a given user selection, which is solved by a numerical method. The ultimate user selection and bandwidth allocation are obtained by iteratively compressing the model and solving these two subproblems. Simulation results show that the proposed FL algorithm can improve the accuracy of object recognition by up to 16.7% and improve the number of selected users by up to 68.7%, compared to a conventional FL algorithm using only RF.

show abstract

Ternary Compression for Communication-Efficient Federated Learning

Cited by 128 publications

References 47 publications

CFD: Communication-Efficient Federated Distillation via Soft-Label Quantization and Delta Coding

CFD: Communication-Efficient Federated Distillation via Soft-Label Quantization and Delta Coding

A federated data-driven evolutionary algorithm for expensive multi-/many-objective optimization

Wireless Network Optimization for Federated Learning with Model Compression in Hybrid VLC/RF Systems

Contact Info

Product

Resources

About