An Efficient Multi-Model Training Algorithm for Federated Learning

Li, Cong; Li, Chunxi; Zhao, Yongqiang; Zhang, Baoxian; Li, Cheng

doi:10.1109/globecom46510.2021.9685230

Cited by 7 publications

(5 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, the higher throughput brought by a larger batch size does not necessarily shorten training time. The result of the relationship between convergence efficiency and batch size also matches recent study [58].…”

Section: Ablation Studysupporting

confidence: 88%

Fold3D: Rethinking and Parallelizing Computational and Communicational Tasks in the Training of Large DNN Models

Zhao

Qing

et al. 2023

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Training a large DNN (e.g., GPT3) efficiently on commodity clouds is challenging even with the latest 3D parallel training systems (e.g., Megatron v3.0). In particular, along the pipeline parallelism dimension, computational tasks that produce a whole DNN's gradients with multiple input batches should be concurrently activated; along the data parallelism dimension, a set of heavy-weight communications (for aggregating the accumulated outputs of computational tasks) is inevitably serialized after the pipelined tasks, undermining the training performance (e.g., in Megatron, data parallelism caused all GPUs idle for over 44% of the training time) over commodity cloud networks. To deserialize these communicational and computational tasks, we propose the AIAO scheduling (for 3D parallelism) which slices a DNN into multiple segments, so that the computational tasks processing the same DNN segment can be scheduled together, and the communicational tasks that synchronize this segment can be launched and overlapped (deserialized) with other segments' computational tasks. We realized this idea in our FOLD3D training system. Extensive evaluation shows FOLD3D eliminated most of the all-GPU 44% idle time in Megatron (caused by data parallelism), leading to 25.2%-42.1% training throughput improvement compared to four notable baselines over various settings; FOLD3D's high performance scaled to many GPUs.

show abstract

Section: Ablation Studysupporting

confidence: 88%

Fold3D: Rethinking and Parallelizing Computational and Communicational Tasks in the Training of Large DNN Models

Zhao

Qing

et al. 2023

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…Training multiple models in a federated setting has been explored in [16] also. However, the setting and the model training methodology are different from ours [3].…”

Section: Related Workmentioning

confidence: 99%

“…However, the setting and the model training methodology are different from ours [3]. In [16], client distribution among models has been approached as an optimization problem and a heuristic algorithm has been proposed. However, neither an optimal algorithm is provided nor any convergence guarantees.…”

Section: Related Workmentioning

confidence: 99%

Multi-Model Federated Learning with Provable Guarantees

Bhuyan¹,

Moharir²,

Joshi³

2022

Preprint

View full text Add to dashboard Cite

Federated Learning (FL) is a variant of distributed learning where edge devices collaborate to learn a model without sharing their data with the central server or each other. We refer to the process of training multiple independent models simultaneously in a federated setting using a common pool of clients as multi-model FL. In this work, we propose two variants of the popular FedAvg algorithm for multi-model FL, with provable convergence guarantees. We further show that for the same amount of computation, multi-model FL can have better performance than training each model separately. We supplement our theoretical results with experiments in strongly convex, convex, and non-convex settings.

show abstract

“…The idea is that these two approaches together will generate a generic model that can be used globally. Another new area of research is multi-model federated learning [29][30][31].…”

Section: Motivation 121 Problem Statementmentioning

confidence: 99%

“…and Moharir [30] proposed multi-model federated learning by simultaneously training multiple uncorrelated models in federated learning. The researchers in [29,31], also worked on multi-model federated learning. As pointed out in Chapter 1, research on multi-modality federated learning is extremely limited [25][26][27][28].…”

Section: Multi-modality Federated Learningmentioning

confidence: 99%

Mutli-Modality Federate Learning with Multi-Source Data for Forest Fire Prediction

View full text Add to dashboard Cite

Forest fires result in significant destruction of natural resources and human lives.They are commonly caused by humans or naturally occurring phenomena like lightning strikes. They may start due to lightning when necessary climatic conditions prevail for a the fire to ignite. The available forest fire data contains both fire and non-fire data and is highly imbalanced. To overcome this, the thesis provides a spatio-temporal agnostic subsampling (STAS) framework to subsample the highly imbalanced forest fire data. The proposed framework also works with limited variations between fire and non-fire data. Apart from lightning, weather and drought conditions are also significant factors in forest fires. Forest fire prediction which does not take these into consideration, may not be accurate. Therefore, the thesis is also focused on building and applying a multi-modality modeling approach for predicting forest fires. The multi-modality model is built using available datasets on forest fires, lightning, weather, and hydrometrics to predict the probability of forest fires. Using the proposed STAS framework and multi-modality model, the research aimed to generate a generalized model by applying multi-modality federated learning to predict forest fires independent of geographic location without the need for calibration.This research was able to successfully work with federated learning framework and multi-modality model to obtain high F 1Score (> 0.9) and R 2 Score (> 0.8).

show abstract

An Efficient Multi-Model Training Algorithm for Federated Learning

Cited by 7 publications

References 6 publications

Fold3D: Rethinking and Parallelizing Computational and Communicational Tasks in the Training of Large DNN Models

Fold3D: Rethinking and Parallelizing Computational and Communicational Tasks in the Training of Large DNN Models

Multi-Model Federated Learning with Provable Guarantees

Mutli-Modality Federate Learning with Multi-Source Data for Forest Fire Prediction

Contact Info

Product

Resources

About