Robust Coreset Construction for Distributed Machine Learning

Lu, Hanlin; Li, Ming-Ju; He, Ting; Wang, Shiqiang; Narayanan, Vijaykrishnan; Chan, Kevin

doi:10.1109/globecom38437.2019.9013625

Cited by 8 publications

(2 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To solve the problem defined in (1)- (2), in this section we propose L-FGADMM, by extending GADMM proposed in our prior work [8]. Following GADMM (see Fig.…”

Section: Proposed Algorithm: L-fgadmmmentioning

confidence: 99%

See 1 more Smart Citation

Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning

Elgabli

Park

Bedi

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

This article proposes a communication-efficient decentralized deep learning algorithm, coined layer-wise federated group ADMM (L-FGADMM). To minimize an empirical risk, every worker in L-FGADMM periodically communicates with two neighbors, in which the periods are separately adjusted for different layers of its deep neural network. A constrained optimization problem for this setting is formulated and solved using the stochastic version of GADMM proposed in our prior work. Numerical evaluations show that by less frequently exchanging the largest layer, L-FGADMM can significantly reduce the communication cost, without compromising the convergence speed. Surprisingly, despite less exchanged information and decentralized operations, intermittently skipping the largest layer consensus in L-FGADMM creates a regularizing effect, thereby achieving the test accuracy as high as federated learning (FL), a baseline method with the entire layer consensus by the aid of a central entity.Layer-wise Federated GADMM (L-FGADMM). To bridge the gap between FL and GADMM, in this article we propose L-FGADMM, by integrating the periodic communication and random data sampling properties of FL into GADMM under a deep NN architecture. To further improve communication efficiency, as illustrated in Fig. 1c, L-GADMM applies a different communication period to each layer. By exchanging the largest layer 2x less frequently than the other layers, our results show that L-FGADMM achieves the same test accuracy while saving 48.8% and 60.8% average communication cost, compared to the case using the same communication period for all layers and FL, respectively. RelatedWorks. Towards improving communication efficiency of distributed ML, under centralized ML, the number of communication rounds can be reduced by collaboratively adjusting the training momentum [11], [12]. On the other hand, the number of communication links can be decreased by collecting model updates until a time deadline [13], upon the values sufficiently changed from the preceding updates [14], [15], or based on channel conditions [16]-[18]. Furthermore, the communication payload can be compressed by 1-bit gradient quantization [19], multi-bit gradient quantization [15], or weight quantization with random rotation [20]. Alternatively, instead of model parameters, model outputs can be exchanged for large models via knowledge distillation [21], [22]. Similar principles are applicable for communicationefficient decentralized ML. Without any central entity, communication payload sizes can be reduced by a quantized weight gossiping algorithm [23], ignoring communication link reduction.Alternatively, the number of communication links and rounds can be decreased using GADMM proposed in our prior work [8]. Furthermore, by integrating stochastic quantization into GADMM, quantized GADMM (Q-GADMM) was proposed to reduce communication rounds, links, and payload sizes altogether [10]. To achieve the same goals, instead of quantization as in Q-GADMM, L-FGADMM applies a layerwise federation to GADMM...

show abstract

“…To solve the problem defined in (1)- (2), in this section we propose L-FGADMM, by extending GADMM proposed in our prior work [8]. Following GADMM (see Fig.…”

Section: Proposed Algorithm: L-fgadmmmentioning

confidence: 99%

“…Interest in data-driven machine learning (ML) is on the rise, but difficulties in securing data still remain [1], [2]. Mission critical applications aggravate this challenge, which require a large volume of up-to-date data for timely coping with local environments even under extreme events.…”

Section: Introductionmentioning

confidence: 99%

Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning

Elgabli

Park

Bedi

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

Coreset-Based Data Reduction for Machine Learning at the Edge

Lu,

He,

Wang

2023

Artificial Intelligence for Edge Computing

View full text Add to dashboard Cite

Robust Coreset Construction for Distributed Machine Learning

et al. 2019

2019 IEEE Global Communications Conference (GLOBECOM)

Self Cite

View full text Add to dashboard Cite

Motivated by the need of solving machine learning problems over distributed datasets, we explore the use of coreset to reduce the communication overhead. Coreset is a summary of the original dataset in the form of a small weighted set in the same sample space. Compared to other data summaries, coreset has the advantage that it can be used as a proxy of the original dataset, potentially for different applications. However, existing coreset construction algorithms are each tailor-made for a specific machine learning problem. Thus, to solve different machine learning problems, one has to collect coresets of different types, defeating the purpose of saving communication overhead. We resolve this dilemma by developing coreset construction algorithms based on k-means/median clustering, that give a guaranteed approximation for a broad range of machine learning problems with sufficiently continuous cost functions. Through evaluations on diverse datasets and machine learning problems, we verify the robust performance of the proposed algorithms.

show abstract

Robust Coreset Construction for Distributed Machine Learning

Cited by 8 publications

References 36 publications

Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning

Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning

Coreset-Based Data Reduction for Machine Learning at the Edge

Robust Coreset Construction for Distributed Machine Learning

Contact Info

Product

Resources

About