Scaling Federated Learning for Fine-Tuning of Large Language Models

Hilmkil, Agrin; Callh, Sebastian; Barbieri, Matteo; Sütfeld, Leon René; Zec, Edvin Listo; Mogren, Olof

doi:10.1007/978-3-030-80599-9_2

Cited by 20 publications

(8 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Comparison of machine learning models with salient features is given in Table I. Federated Learning Network (FLN) has also been adopted to orchestrate various mobile devices across the world for training language models with BERT [7]. All such mobile devices are owned by different users and then connected to multiple types of links such as WiFi, mobile network, etc.…”

Section: Federated Model and Critical Analysismentioning

confidence: 99%

Federated Learning and its Applications for Security and Communication

Asif¹,

Karim²,

Kausar³

2022

IJACSA

View full text Add to dashboard Cite

The not so long ago, Artificial Intelligence (AI) has revolutionized our life by giving rise to the idea of self-learning in different environments. Amongst its different variants, Federated Learning (FL) is a novel approach that relies on decentralized communication data and its associated training. While reducing the amount of data acquired from users, federated learning derives the benefits of popular machine learning techniques, it brings learning to the edge or directly on-device. FL, frequently referred to as a new dawn in AI, is still in its early stages and is yet to garner widespread acceptance, owing to its (unknown) security and privacy implications. In this paper, we give an illustrative explanation of FL techniques, communication, and applications with privacy as well as security issues. According to our findings, there are fewer privacy-specific dangers linked with FL than security threats. We conclude the paper with the challenges of FL with special emphases on security.

show abstract

Section: Federated Model and Critical Analysismentioning

confidence: 99%

Federated Learning and its Applications for Security and Communication

Asif¹,

Karim²,

Kausar³

2022

IJACSA

View full text Add to dashboard Cite

show abstract

“…Specifically, we examine cross-device FL (Kairouz et al, 2021b), where local clients are edge devices with limited resources and computing power, which can number in the millions. Previous works on language modeling in cross-device FL often use small recurrent-based models of less than 10M parameters (Hard et al, 2018b;, while more recent works leverage a variety of efficient techniques for training larger Transformer-based models (Hilmkil et al, 2021;Ro et al, 2022). In this work, we investigate modular strategies applicable to various model architectures for improving training of both small and large models in cross-device FL.…”

Section: Introductionmentioning

confidence: 99%

Scaling Language Model Size in Cross-Device Federated Learning

Ro¹,

Breiner²,

McConnaughey³

et al. 2022

Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022)

View full text Add to dashboard Cite

Cross-device federated learning (FL) is a technique that trains a model on data distributed across typically millions of edge devices without data leaving the devices. SGD is the standard client optimizer for on device training in cross-device FL, favored for its memory and computational efficiency. However, in centralized training of neural language models, adaptive optimizers are preferred as they offer improved stability and performance. In light of this, we ask if language models can be modified such that they can be efficiently trained with SGD client optimizers and answer this affirmatively.We propose a scale-invariant Coupled Input Forget Gate (SI CIFG) recurrent network by modifying the sigmoid and tanh activations in the recurrent cell and show that this new model converges faster and achieves better utility than the standard CIFG recurrent model in cross-device FL in large scale experiments. We further show that the proposed scale invariant modification also helps in federated learning of larger transformer models. Finally, we demonstrate the scale invariant modification is also compatible with other non-adaptive algorithms. Particularly, our results suggest an improved privacy utility trade-off in federated learning with differential privacy.

show abstract

“…[12] confirms that it is possible to both pre-train and fine-tune BERT models in a federated manner using clinical texts from different silos without moving the data. [13] provides an overview of the applicability of the federated learning to Transformer-based language models. [14] firstly applied federated learning to Transformer-based Neural Machine Translation to avoid sharing customers' chat recording with the server.…”

Section: Introductionmentioning

confidence: 99%

Federated Split BERT for Heterogeneous Text Classification

Li¹,

Si²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pre-trained BERT models have achieved impressive performance in many natural language processing (NLP) tasks. However, in many real-world situations, textual data are usually decentralized over many clients and unable to be uploaded to a central server due to privacy protection and regulations. Federated learning (FL) enables multiple clients collaboratively to train a global model while keeping the local data privacy. A few researches have investigated BERT in federated learning setting, but the problem of performance loss caused by heterogeneous (e.g., non-IID) data over clients remain under-explored. To address this issue, we propose a framework, FedSplitBERT, which handles heterogeneous data and decreases the communication cost by splitting the BERT encoder layers into local part and global part. The local part parameters are trained by the local client only while the global part parameters are trained by aggregating gradients of multiple clients. Due to the sheer size of BERT, we explore a quantization method to further reduce the communication cost with minimal performance loss. Our framework is ready-to-use and compatible to many existing federated learning algorithms, including FedAvg, FedProx and FedAdam. Our experiments verify the effectiveness of the proposed framework, which outperforms baseline methods by a significant margin, while FedSplitBERT with quantization can reduce the communication cost by 11.9×.

show abstract

Scaling Federated Learning for Fine-Tuning of Large Language Models

Cited by 20 publications

References 4 publications

Federated Learning and its Applications for Security and Communication

Federated Learning and its Applications for Security and Communication

Scaling Language Model Size in Cross-Device Federated Learning

Federated Split BERT for Heterogeneous Text Classification

Contact Info

Product

Resources

About