Federated Reconstruction: Partially Local Federated Learning

Singhal, Karan; Sidahmed, Hakim; Garrett, Zachary; Wu, Shanshan; Rush, Keith; Prakash, Sushant

doi:10.48550/arxiv.2102.03448

Cited by 4 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Splitting the entire model into a shared part and a local part is another natural approach to personalization. Unlike the stateful algorithms mentioned in Section 7.5.1, Singhal et al [230] proposed an algorithm where the local part is reconstructed locally each time a client participates in a round, and showed that the proposed algorithm also fits in the meta-learning framework.…”

Section: Algorithms That Do Not Require Client-side State or Identifiermentioning

confidence: 99%

A Field Guide to Federated Optimization

Wang¹,

Charles²,

Xu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection. The distributed learning process can be formulated as solving federated optimization problems, which emphasize communication efficiency, data heterogeneity, compatibility with privacy and system requirements, and other constraints that are not primary considerations in other problem settings. This paper provides recommendations and guidelines on formulating, designing, evaluating and analyzing federated optimization algorithms through concrete examples and practical implementation, with a focus on conducting effective simulations to infer real-world performance. The goal of this work is not to survey the current literature, but to inspire researchers and practitioners to design federated learning algorithms that can be used in various practical applications.

show abstract

Section: Algorithms That Do Not Require Client-side State or Identifiermentioning

confidence: 99%

A Field Guide to Federated Optimization

Wang¹,

Charles²,

Xu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Closely related to federated dropout, ordered dropout Horvath et al [2021] extracts submodels from a main model, and adapts the computation and communication costs to the edge device capabilities. Singhal et al [2021] propose a method to partially reconstruct a model on the edge devices, which results in a reduced communication cost. The reconstructed parameters are local to the clients, and never sent to the server.…”

Section: Related Workmentioning

confidence: 99%

Efficient and Private Federated Learning with Partially Trainable Networks

Sidahmed¹,

Xu²,

Garg³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning is used for decentralized training of machine learning models on a large number (millions) of edge mobile devices. It is challenging because mobile devices often have limited communication bandwidth and local computation resources. Therefore, improving the efficiency of federated learning is critical for scalability and usability. In this paper, we propose to leverage partially trainable neural networks, which freeze a portion of the model parameters during the entire training process, to reduce the communication cost with little implications on model performance. Through extensive experiments, we empirically show that Federated learning of Partially Trainable neural networks (FedPT) can result in superior communication-accuracy trade-offs, with up to 46× reduction in communication cost, at a small accuracy cost. Our approach also enables faster training, with a smaller memory footprint, and better utility for strong differential privacy guarantees. The proposed FedPT can be particularly interesting for pushing the limitations of overparameterization in on-device learning.

show abstract

“…In recent years, deep learning language models trained on a large of centralized data have achieved an impressive performance in various NLP downstream tasks (e.g., medical case analysis [1], sentiment analysis [2], nextword prediction in mobile keyboards [3]). Such success is mainly due to advanced machine learning techniques and large-scale data collection.…”

Section: Introductionmentioning

confidence: 99%

Differentially Private Recurrent Variational Autoencoder For Text Privacy Preservation

2023

View full text Add to dashboard Cite

Deep learning techniques have been widely used in natural language processing (NLP) tasks and have made remarkable progress. However, training the deep learning model relies on a large amount of data which may involve sensitive information like electronic medical records. The attacker can infer sensitive information from the model, which leads to privacy leakage. To solve this problem, we propose a Differentially Private Recurrent Variational AutoEncoder (DP-RVAE) that can generate simulated data in place of the sensitive dataset to preserve privacy. To generate high utility synthetic text, a part of sensitive text data is employed as the conditional input of the model and uses a dropout and noise perturbing mechanism to preserve differential privacy. In addition, we expand the proposed DP-RVAE to a federated learning setting and design a novel training paradigm for NLP tasks. Specifically, DP-RVAE is deployed to the client-side to train and generate personalized text. These DP-RVAE models would be aggregated and updated through the Federated Optimisation (FedOPT) algorithm so that personal information can be well preserved. We evaluate our proposed DP-RVAE through a text classification task on the Tweets depression sentiment and IMDB reviews datasets. Our DP-RVAE achieves a higher average test accuracy by 5.90% and 3.94% compared to the typical centralized training and federated learning approach, respectively. We also perform the keywords inference attack experiment on the medical description dataset collected from the real world. Compared to the typical differentially private preserving approach, the DP-RVAE decreases by 15.2% in average attack accuracy. The experimental results demonstrate that DP-RVAE can be applied to the NLP models to leverage accuracy while preserving sensitive privacy.

show abstract

Federated Reconstruction: Partially Local Federated Learning

Cited by 4 publications

References 0 publications

A Field Guide to Federated Optimization

A Field Guide to Federated Optimization

Efficient and Private Federated Learning with Partially Trainable Networks

Differentially Private Recurrent Variational Autoencoder For Text Privacy Preservation

Contact Info

Product

Resources

About