Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression

Qiao, Zhefeng; Yu, Xianghao; Zhang, Jun; Letaief, Khaled B.

doi:10.48550/arxiv.2104.12416

Cited by 3 publications

(5 citation statements)

References 13 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The server then averages all received sub-sampled updates to get an estimate of the global model parameters. Additionally, [52] used dual-side low-rank compression to reduce the size of the models in both directions between the server and the nodes. Finally, [36] used a layer-based parameter selection in order to transfer only the important parameters of each model's layer.…”

Section: Updates Compressionmentioning

confidence: 99%

Federated learning for energy constrained devices: a systematic mapping study

Mokadem

Maissa²,

Akkaoui³

2022

Cluster Comput

View full text Add to dashboard Cite

Federated Machine Learning (Fed ML) is a new distributed machine learning technique applied to collaboratively train a global model using clients' local data without transmitting it. Nodes only send parameter updates (e.g., weight updates in the case of neural networks), which are fused together by the server to build the global model. By not divulging node data, Fed ML guarantees its confidentiality, a crucial aspect of network security, which enables it to be used in the context of data-sensitive Internet of Things (IoT) and mobile applications, such as smart geo-location and the smart grid. However, most IoT devices are particularly energy constrained, which raises the need to optimize the Fed ML process for efficient training tasks and optimized power consumption. In this paper, we conduct, to the best of our knowledge, the first Systematic Mapping Study (SMS) on FedML optimization techniques for energy-constrained IoT devices. From a total of more than 800 papers, we select 67 that satisfy our criteria and give a structured overview of the field using a set of carefully chosen research questions. Finally, we attempt to provide an analysis of the energy-constrained Fed ML state of the art and try to outline some potential recommendations for the research community.

show abstract

Section: Updates Compressionmentioning

confidence: 99%

Federated learning for energy constrained devices: a systematic mapping study

Mokadem

Maissa²,

Akkaoui³

2022

Cluster Comput

View full text Add to dashboard Cite

show abstract

“…FL enables a multitude of participants to construct a joint model without sharing their private training data [4,22,23,25]. Some recent work focus on compressing the parameters or transmitting partial network for efficient transmission [2,20,28,30]. However, they do not reduce the computation overhead on edge.…”

Section: Related Workmentioning

confidence: 99%

“…Furthermore, directly training the low-rank model from scratch [12] results in the performance drop. FedDLR [28] factorizes the server model and recovers the low-rank model on the clients, reducing the communication cost while increasing the local computation cost. Pufferfish [32] improves the performance of the low-rank model by training a hybrid network and warm-up.…”

Section: Related Workmentioning

confidence: 99%

FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization

Yao¹,

Pan²,

Wan³

et al. 2021

Preprint

View full text Add to dashboard Cite

The underlying assumption of recent federated learning (FL) paradigms is that local models usually share the same network architecture as the global model, which becomes impractical for mobile and IoT devices with different setups of hardware and infrastructure. A scalable federated learning framework should address heterogeneous clients equipped with different computation and communication capabilities. To this end, this paper proposes FEDHM, a novel federated model compression framework that distributes the heterogeneous low-rank models to clients and then aggregates them into a global full-rank model. Our solution enables the training of heterogeneous local models with varying computational complexities and aggregates a single global model. Furthermore, FEDHM not only reduces the computational complexity of the device, but also reduces the communication cost by using low-rank models. Extensive experimental results demonstrate that our proposed FEDHM outperforms the current pruning-based FL approaches in terms of test Top-1 accuracy (4.6% accuracy gain on average), with smaller model size (1.5× smaller on average) under various heterogeneous FL settings.

show abstract

“…In this regard, an important task in SWIPT-based FL is to optimize the portion of harvested energy allocated to communication with the edge server and local computation, respectively. In addition, the integration of SWIPT and conventional energy-saving techniques in FL, e.g., model compression, adaptive transmission, and hierarchical FL [115], needs further investigation.…”

Section: Mobile Edge Computing and Federated Learningmentioning

confidence: 99%

“…Conventional design methodologies based on mathematical optimization may not be directly applicable to large-scale SWIPT networks as the complexity of the optimal designs often scales exponentially with the network size. As a remedy, ML is a promising tool as it can be used to optimize large-scale systems without relying on analytical models [115], [116]. Recently, several papers have exploited ML for solving communication-centric resource allocation design problems in conventional wireless networks [117]- [119].…”

Section: E Machine Learning-based Designmentioning

confidence: 99%

Resource Allocation for Simultaneous Wireless Information and Power Transfer Systems: A Tutorial Overview

Wei¹,

Yu²,

Ng³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Over the last decade, simultaneous wireless information and power transfer (SWIPT) has become a practical and promising solution for connecting and recharging batterylimited devices, thanks to significant advances in low-power electronics technology and wireless communications techniques. To realize the promised potentials, advanced resource allocation design plays a decisive role for revealing, understanding, and exploiting the intrinsic rate-energy tradeoff capitalizing on the dual use of radio frequency (RF) signals for wireless charging and communication. In this paper, we provide a comprehensive tutorial overview of SWIPT from the perspective of resource allocation design. The fundamental concepts, system architectures, and RF energy harvesting (EH) models are introduced. In particular, three commonly adopted EH models, namely the linear EH model, the nonlinear saturation EH model, and the nonlinear circuit-based EH model are characterized and discussed. Then, for a typical wireless system setup, we establish a generalized resource allocation design framework which subsumes conventional resource allocation design problems as special cases. Subsequently, we elaborate on relevant tools from optimization theory and exploit them for solving representative resource allocation design problems for SWIPT systems with and without perfect channel state information (CSI) available at the transmitter, respectively. The associated technical challenges and insights are also highlighted. Furthermore, we discuss several promising and exciting future research directions for resource allocation design for SWIPT systems intertwined with cuttingedge communication technologies, such as intelligent reflecting surfaces, unmanned aerial vehicle, mobile edge computing, federated learning, and machine learning.

show abstract

Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression

Cited by 3 publications

References 13 publications

Federated learning for energy constrained devices: a systematic mapping study

Federated learning for energy constrained devices: a systematic mapping study

FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization

Resource Allocation for Simultaneous Wireless Information and Power Transfer Systems: A Tutorial Overview

Contact Info

Product

Resources

About