Private Weighted Random Walk Stochastic Gradient Descent

Abstract

We consider a decentralized learning setting in which data is distributed over nodes in a graph.The goal is to learn a global model on the distributed data without involving any central entity that needs to be trusted. While gossip-based stochastic gradient descent (SGD) can be used to achieve this learning objective, it incurs high communication and computation costs, since it has to wait for all the local models at all the nodes to converge. To speed up the convergence, we propose instead to study random walk based SGD in which a global model is updated based on a random walk on the graph.We propose two algorithms based on two types of random walks that achieve, in a decentralized way, uniform sampling and importance sampling of the data. We provide a non-asymptotic analysis on the rate of convergence, taking into account the constants related to the data and the graph. Our numerical results show that the weighted random walk based algorithm has a better performance for high-variance data. Moreover, we propose a privacy-preserving random walk algorithm that achieves local differential privacy based on a Gamma noise mechanism that we propose. We also give numerical results on the convergence of this algorithm and show that it outperforms additive Laplace-based privacy mechanisms.

Walk for Learning: A Random Walk Approach for Federated Learning from Heterogeneous Data

“…In [13], the authors studied the convergence of random walks learning for the alternating direction method of multipliers (ADMM). In [21], the paper proposes to improve the convergence guarantees by designing a weighted random walk that accounts for the importance of the local data to speed up convergence. An asymptotic fundamental bound on the convergence rate of these algorithms was proven by [22] and it approaches O(1/ √ k) under convexity and bounded-gradient assumptions.…”

“…The goal of a Gossip algorithm is to ensure that all nodes, and not just the PS, learn the global model and assume convergence once a consensus is reached. Hence, it is less efficient in terms of computations and communication costs [21].…”

“…Assumption 3 (Bounded Gradient). There exits a constant D such that, ∀i ∈ [V ] and ∀w ∈ W, we have ∇F i (w) 2 2 ≤ D. The last assumption is actually a result that follows from the functions F i 's being convex on a closed bounded subset W ⊂ R. A complementary proof can be found in [21].…”

“…In order to prove Lemma 4 and Theorem 1, we present some technical results that we use in the proof. The proof techniques are essentially inspired by the work of [21] and theya are adapted to the assumptions and setting of this work.…”

Stability and Generalization for Markov Chain Stochastic Gradient Methods

“…Markov chain naturally appears in many important problems, such as decentralized consensus optimization, which finds applications in various areas including wireless sensor networks, smart grid implementations and distributed statistical learning [4,14,23,48,50,58,61,63] as well as pairwise learning [78] which instantiates AUC maximization [1,29,46,81,87] and metric learning [35,75,76,79]. A common example is a distributed system in which each node stores a subset of the whole data, and one aims to train a global model based on these data.…”