Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent

Lian, Xiangru; Zhang, Ce; Zhang, Huan; Hsieh, Cho‐Jui; Zhang, Wei; Liu, Ji

doi:10.48550/arxiv.1705.09056

Cited by 25 publications

(38 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Decentralized learning has many advantages over centralized learning. In [3], Lian et al rigorously proved that a decentralized algorithm has lower communication complexity and possesses the same convergence rate as those under a centralized parameter-server model. Furthermore, a centralized topology might not hold in a decentralized network where no one can be trusted enough to act as a parameter server.…”

Section: Decentralized Learningmentioning

confidence: 99%

“…Such a decentralized network is frequently adopted in ad hoc networks, edge computing, Internet-of-Things (IoT), decentralized applications (Dapp), etc. It can greatly unleash the potential for building large-scale (even worldwide) machine learning models that can reasonably and fully maximize the utilization of computational resources [3]. Besides, devices such as mobile Fig.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SPDL: Blockchain-secured and Privacy-preserving Decentralized Learning

Xu¹,

Zou²,

Cheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Decentralized learning involves training machine learning models over remote mobile devices, edge servers, or cloud servers while keeping data localized. Even though many studies have shown the feasibility of preserving privacy, enhancing training performance or introducing Byzantine resilience, but none of them simultaneously considers all of them. Therefore we face the following problem: how can we efficiently coordinate the decentralized learning process while simultaneously maintaining learning security and data privacy? To address this issue, in this paper we propose SPDL, a blockchain-secured and privacy-preserving decentralized learning scheme. SPDL integrates blockchain, Byzantine Fault-Tolerant (BFT) consensus, BFT Gradients Aggregation Rule (GAR), and differential privacy seamlessly into one system, ensuring efficient machine learning while maintaining data privacy, Byzantine fault tolerance, transparency, and traceability. To validate our scheme, we provide rigorous analysis on convergence and regret in the presence of Byzantine nodes. We also build a SPDL prototype and conduct extensive experiments to demonstrate that SPDL is effective and efficient with strong security and privacy guarantees.

show abstract

Section: Decentralized Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

SPDL: Blockchain-secured and Privacy-preserving Decentralized Learning

Xu¹,

Zou²,

Cheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Different aggregation techniques that compound the local model updates from all the clients participating in the training cycle have been presented in this area [23]. Aggregation algorithms may be divided into three types: centralized, hierarchical, and decentralized [24].…”

Section: Aggregation Algorithmsmentioning

confidence: 99%

Secure Multi-Party Computation based Privacy Preserving Data Analysis in Healthcare IoT Systems

ŞAHİNBAŞ¹,

Çatak²

2021

Preprint

View full text Add to dashboard Cite

Recently, many innovations have been experienced in healthcare by rapidly growing Internet-of-Things (IoT) technology that provides significant developments and facilities in the health sector and improves daily human life. The IoT bridges people, information technology and speed up shopping. For these reasons, IoT technology has started to be used on a large scale. Thanks to the use of IoT technology in health services, chronic disease monitoring, health monitoring, rapid intervention, early diagnosis and treatment, etc. facilitates the delivery of health services. However, the data transferred to the digital environment pose a threat of privacy leakage. Unauthorized persons have used them, and there have been malicious attacks on the health and privacy of individuals. In this study, it is aimed to propose a model to handle the privacy problems based on federated learning. Besides, we apply secure multi party computation. Our proposed model presents an extensive privacy and data analysis and achieve high performance.

show abstract

“…Nothing is known about the slowly-decaying nonsquare-summable ones, which are generally preferable since they give similar benefits as constant stepsizes and, often, also guarantee convergence. Note that such issues also plague much of the distributed stochastic optimization literature Yuan et al [2016], Sun et al [2019], Lian et al [2017], Koloskova et al [2020], , Pu and Nedić [2020].…”

Section: Related Workmentioning

confidence: 99%

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

Thoppe¹,

Kumar²

2021

Preprint

View full text Add to dashboard Cite

In Multi-Agent Reinforcement Learning (MARL), multiple agents interact with a common environment, as also with each other, for solving a shared problem in sequential decision-making. It has wide-ranging applications in gaming, robotics, finance, etc. In this work, we derive a novel law of iterated logarithm for a family of distributed nonlinear stochastic approximation schemes that is useful in MARL. In particular, our result describes the convergence rate on almost every sample path where the algorithm converges. This result is the first of its kind in the distributed setup and provides deeper insights than the existing ones, which only discuss convergence rates in the expected or the CLT sense. Importantly, our result holds under significantly weaker assumptions: neither the gossip matrix needs to be doubly stochastic nor the stepsizes square summable. As an application, we show that, for the stepsize n −γ with γ ∈ (0, 1), the distributed TD(0) algorithm with linear function approximation has a convergence rate of O( √ n −γ ln n) a.s.; for the 1/n type stepsize, the same is O( √ n −1 ln ln n) a.s. These decay rates do not depend on the graph depicting the interactions among the different agents.Preprint. Under review.

show abstract

Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent

Cited by 25 publications

References 40 publications

SPDL: Blockchain-secured and Privacy-preserving Decentralized Learning

SPDL: Blockchain-secured and Privacy-preserving Decentralized Learning

Secure Multi-Party Computation based Privacy Preserving Data Analysis in Healthcare IoT Systems

A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning

Contact Info

Product

Resources

About