Time-Correlated Sparsification for Efficient Over-the-Air Model Aggregation in Wireless Federated Learning

Sun, Yuxuan; Zhou, Sheng; Niu, Zhisheng; Gündüz, Deniz

doi:10.48550/arxiv.2202.08420

Cited by 3 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…. , | Ṽ |}, required subpacket (i.e., V (i) = P ( Ṽ (i))) from the set Ṽ , database n picks the column Ṽ (i) of the permutation-reversing matrix R n given in (10) indicated by R n (:, Ṽ (i)) and calculates the corresponding query given by,…”

Section: B Reading Phase At Time Tmentioning

confidence: 99%

Private Federated Submodel Learning with Sparsification

Sajani¹,

Ulukuş²

2022

Preprint

View full text Add to dashboard Cite

We investigate the problem of private read update write (PRUW) in federated submodel learning (FSL) with sparsification. In FSL, a machine learning model is divided into multiple submodels, where each user updates only the submodel that is relevant to the user's local data. PRUW is the process of privately performing FSL by reading from and writing to the required submodel without revealing the submodel index or the values of updates to the databases. Sparsification is a widely used concept in learning, where the users update only a small fraction of parameters to reduce the communication cost. Revealing the coordinates of these selected (sparse) updates leaks privacy of the user. We show how PRUW in FSL can be performed with sparsification. We propose a novel scheme which privately reads from and writes to arbitrary parameters of any given submodel, without revealing the submodel index, values of the updates, or the coordinates of the sparse updates, to databases. The proposed scheme achieves significantly lower reading and writing costs compared to what is achieved without sparsification.

show abstract

Section: B Reading Phase At Time Tmentioning

confidence: 99%

Private Federated Submodel Learning with Sparsification

Sajani¹,

Ulukuş²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We note that the ITU standard of 5G uplink user experienced data rate is only 50 Mb/s [44], and even the 6G uplink user experienced data rate at Gbit/s level may not sufficiently support such huge uploading requirements of regular FL. Additionally, to our best knowledge, existing works on improving FL efficiency (such as FedAvg [4], sparsification [45], [46] and quantization [47], [48] with or without error feedback [49], [50], federated distillation [51]- [53], pruning [54], [55] or partially trainable network [56], [57], and over-the-air computation [58]- [60]) still consider the transmission of gradient updates and can achieve a relatively limited reduction in payload and experience degradation in performance. For instance, the payload reduction is of only two orders of magnitude of the original payload on the same CIFAR-10 dataset [61], [62].…”

Section: A Contributionsmentioning

confidence: 99%

Communication-Efficient and Privacy-Preserving Feature-based Federated Transfer Learning

Wang

Gursoy

Velipasalar

2022

GLOBECOM 2022 - 2022 IEEE Global Communications Conference

View full text Add to dashboard Cite

Federated learning (FL) is an emerging paradigm for training deep neural networks (DNNs) in distributed manners. Current FL approaches all suffer from high communication overhead and information leakage. In this work, we present a federated learning algorithm based on evolution strategies (FedES), a zeroth-order training method. Instead of transmitting model parameters, FedES only communicates loss values, and thus has very low communication overhead. Moreover, a third party is unable to estimate gradients without knowing the pre-shared seed, which protects data privacy. Experimental results demonstrate FedES can achieve the above benefits while keeping convergence performance the same as that with back propagation methods.

show abstract

“…Apart from the privacy leakage, another drawback of FL is the large communication cost incurred by sharing model parameters and updates with millions of users in multiple rounds. Some of the solutions to this problem include, gradient quantization [25][26][27][28], federated submodel learning (FSL) [29][30][31][32][33][34][35][36][37], and gradient sparsification [38][39][40][41][42][43][44][45]. In gradient quantization, the values of the gradients are quantized and represented with a fewer number of bits.…”

Section: Introductionmentioning

confidence: 99%

Private Federated Submodel Learning with Sparsification

Sajani

Ulukuş

2022

2022 IEEE Information Theory Workshop (ITW)

View full text Add to dashboard Cite

In federated learning (FL), a machine learning (ML) model is collectively trained by a large number of users, using their private data in their local devices. With top r sparsification in FL, the users only upload the most significant r fraction of updates, and the servers only send the most significant r ′ fraction of parameters to the users in order to reduce the communication cost. However, the values and the indices of the sparse updates leak information about the users' private data. In this work, we consider an FL setting where N non-colluding databases store the model to be trained, from which the users download and update sparse parameters privately, without revealing the values of the updates or their indices to the databases. We propose four schemes with different properties to perform this task while achieving the minimum communication costs, and show that the information theoretic privacy of both values and positions of the sparse updates can be guaranteed. This is achieved at a considerable storage cost, though. To alleviate this, we generalize the schemes in such a way that the storage cost is reduced at the expense of a certain amount of information leakage, using a model segmentation mechanism. In general, we provide the tradeoff between communication cost, storage cost and information leakage in private FL with top r sparsification.

show abstract

Time-Correlated Sparsification for Efficient Over-the-Air Model Aggregation in Wireless Federated Learning

Cited by 3 publications

References 12 publications

Private Federated Submodel Learning with Sparsification

Private Federated Submodel Learning with Sparsification

Communication-Efficient and Privacy-Preserving Feature-based Federated Transfer Learning

Private Federated Submodel Learning with Sparsification

Contact Info

Product

Resources

About