ClusterGrad: Adaptive Gradient Compression by Clustering in Federated Learning

Cui, Laizhong; Su, Xiaoxin; Zhou, Yipeng; Zhang, Lei

doi:10.1109/globecom42002.2020.9322527

Cited by 10 publications

(3 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The DGC algorithm proposed in [24] discarded 99.9% of insignificant model updates for uploading on distributed machine learning workers, and different methods were designed to maintain the performance of the model. ClusterGrad [25] was based on clustering to adaptively filter and quantify important gradients according to the distribution of model updates. Li et al [26] took into account the heterogeneous resources on devices in FL.…”

Section: Communication Compression In Flmentioning

confidence: 99%

Optimal Rate Adaption in Federated Learning with Compressed Communications

Cui

Zhou

et al. 2022

IEEE INFOCOM 2022 - IEEE Conference on Computer Communications

Self Cite

View full text Add to dashboard Cite

Recently, blockchain-based federated learning (BFL) has attracted intensive research attention due to that the training process is auditable and the architecture is serverless avoiding the single point failure of the parameter server in vanilla federated learning (VFL). Nevertheless, BFL tremendously escalates the communication traffic volume because all local model updates (i.e., changes of model parameters) obtained by BFL clients will be transmitted to all miners for verification and to all clients for aggregation. In contrast, the parameter server and clients in VFL only retain aggregated model updates. Consequently, the huge communication traffic in BFL will inevitably impair the training efficiency and hinder the deployment of BFL in reality. To improve the practicality of BFL, we are among the first to propose a fast blockchain-based communication-efficient federated learning framework by compressing communications in BFL, called BCFL. Meanwhile, we derive the convergence rate of BCFL with non-convex loss. To maximize the final model accuracy, we further formulate the problem to minimize the training loss of the convergence rate subject to a limited training time with respect to the compression rate and the block generation rate, which is a bi-convex optimization problem and can be efficiently solved. To the end, to demonstrate the efficiency of BCFL, we carry out extensive experiments with standard CIFAR-10 and FEMNIST datasets. Our experimental results not only verify the correctness of our analysis, but also manifest that BCFL can remarkably reduce the communication traffic by 95-98% or shorten the training time by 90-95% compared with BFL.

show abstract

Section: Communication Compression In Flmentioning

confidence: 99%

Optimal Rate Adaption in Federated Learning with Compressed Communications

Cui

Zhou

et al. 2022

IEEE INFOCOM 2022 - IEEE Conference on Computer Communications

Self Cite

View full text Add to dashboard Cite

show abstract

“…A first approach aims at reducing the energy spent for the communications between the clients and the central server. Gradient sparsification and gradient quantization techniques (see, e.g., [7,3,6]) have been recently proposed for that purpose. Since compression is a lossy process, the gains in terms of communication costs are usually achieved at the expense of a worse iteration complexity and there is not a good understanding of how these techniques impact the total energy consumption and whether they are worth applying in general.…”

Section: Introductionmentioning

confidence: 99%

Data Summarization for Federated Learning

Devillers,

Brun,

Prabhu

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We explore data summarization techniques as a mean to reduce the energy footprint of Federated Learning (FL). We formulate the problem of selecting a small subset of data points that best represent the gradient of each local dataset as a submodular maximization problem and provide sufficient conditions under which the FL training is guaranteed to converge to the same global model as if the whole local datasets have been used on each client. Experimental results on IID and non-IID datasets show that this approach yields a similar accuracy as training on the full local datasets, but with a significant reduction of runtimes. There is however no clear advantage of data summarization over random sampling.

show abstract

“…Broadly speaking, the main processes of FL can be described as below: One of FL's mainstream scenarios is that the server locates in a data center with good Internet resources, and clients are edge devices (e.g., smartphones, IoTs, or wearable devices) (Konecný et al 2016a,b;McMahan et al 2017;Sattler et al 2020;Smith et al 2017;Zheng et al 2020) which are limited in low bandwidth and unstable network. Existing studies show that residential Internet connections tend to reach far higher download than upload speeds (Goga and Teixeira 2012;Konecný et al 2016a;Kairouz et al 2019;Cui et al 2020;Zhang et al 2021aZhang et al , 2022, thus there exist a serious bottleneck in FL when a huge number of clients send their local model to the server, especially when the uploaded models are the complicated and large-scale convolutional neural networks (CNNs).…”

Section: Introductionmentioning

confidence: 99%

SmartIdx: Reducing Communication Cost in Federated Learning by Exploiting the CNNs Structures

Zou

Zhang

et al. 2022

AAAI

View full text Add to dashboard Cite

Top-k sparsification method is popular and powerful forreducing the communication cost in Federated Learning(FL). However, according to our experimental observation, it spends most of the total communication cost on the index of the selected parameters (i.e., their position informa-tion), which is inefficient for FL training. To solve this problem, we propose a FL compression algorithm for convolution neural networks (CNNs), called SmartIdx, by extending the traditional Top-k largest variation selection strategy intothe convolution-kernel-based selection, to reduce the proportion of the index in the overall communication cost and thusachieve a high compression ratio. The basic idea of SmartIdx is to improve the 1:1 proportion relationship betweenthe value and index of the parameters to n:1, by regarding the convolution kernel as the basic selecting unit in parameter selection, which can potentially deliver more informationto the parameter server under the limited network traffic. Tothis end, a set of rules are designed for judging which kernel should be selected and the corresponding packaging strategies are also proposed for further improving the compressionratio. Experiments on mainstream CNNs and datasets show that our proposed SmartIdx performs 2.5×−69.2× higher compression ratio than the state-of-the-art FL compression algorithms without degrading model performance.

show abstract

ClusterGrad: Adaptive Gradient Compression by Clustering in Federated Learning

Cited by 10 publications

References 8 publications

Optimal Rate Adaption in Federated Learning with Compressed Communications

Optimal Rate Adaption in Federated Learning with Compressed Communications

Data Summarization for Federated Learning

SmartIdx: Reducing Communication Cost in Federated Learning by Exploiting the CNNs Structures

Contact Info

Product

Resources

About