Consensus-Based Cooperative Algorithms for Training Over Distributed Data Sets Using Stochastic Gradients

Abstract

In this paper, distributed algorithms are proposed for training a group of neural networks with private datasets. Stochastic gradients are utilised in order to eliminate the requirement for true gradients. To obtain a universal model of the distributed neural networks trained using local datasets only, consensus tools are introduced to derive the model towards the optimum. Most of the existing works employ diminishing learning rates, which are often slow and impracticable for online learning, while constant learning rates are studied in some recent works, but the principle for choosing the rates is not well established. In this paper, constant learning rates are adopted to empower the proposed algorithms with tracking ability. Under mild conditions, the convergence of the proposed algorithms is established by exploring the error dynamics of the connected agents, which provides an upper bound for selecting the constant learning rates. Performances of the proposed algorithms are analysed with and without gradient noises, in the sense of mean-square-error (MSE). It is proved that the MSE converges with bounded errors determined by the gradient noises, and the MSE converges to zero if the gradient noises are absent. Simulation results are provided to validate the effectiveness of the proposed algorithms.

Distributed stochastic proximal algorithm with random reshuffling for non-smooth finite-sum optimization

“…By summing (26) over i from 0 to n − 1, we obtain an upper bound of the forward deviation V t by Lemma IV.1 that…”

“…The potential bottleneck of the starshaped network lies on the communication traffic jam on the central sever and the performance will be significantly degraded when the network bandwidth is low. To consider a more general distributed network topology without a central server, many distributed stochastic gradient algorithms have been studied [21]- [26] for convex finite-sum optimization. Considering the network communication, some works have studied synchronous and asynchronous distributed stochastic algorithms with diminishing step-sizes over multi-agent networks [22]- [24].…”

“…Considering the network communication, some works have studied synchronous and asynchronous distributed stochastic algorithms with diminishing step-sizes over multi-agent networks [22]- [24]. Furthermore, to avoid the degenerated performance of algorithms with diminishing steps-sizes, [25] and [26] have proposed consensus-based distributed stochastic gradient descent methods with non-diminishing (constant) step-sizes for smooth convex optimization over time-invariant networks. With variance reduction to handle the variance of the local stochastic gradients at each node, some distributed efficient stochastic algorithms with constant step-sizes have been studied for strongly convex smooth optimization over time-invariant undirected networks [27] and time-invariant directed networks [28].…”

Exploiting abstractions for grammar‐based learning of complex multi‐agent behaviours

“…Similarly, techniques have also been investigated to process data in parallel with multiple local models committing updates simultaneously. 30 Consensus-based parallel learning algorithms where multiple training representations are learned in parallel, and information of locally trained parameters are then exchanged to build consensus are examples of such applications. 31 As such, both data-driven domains as well as agent-based domains use parallel and incremental architectures with similar interests of distributing the learning process to overcome the limitations associated with centralised learning.…”

Convergence of the Distributed SG Algorithm Under Cooperative Excitation Condition

“…The stochastic gradient algorithm is widely applied in the area of adaptive control, and also has deep connections with stochastic gradient descent algorithm and its variants which are widely used to deal with optimization problems in machine learning [23]. With the development of sensor networks, the distributed implementations of stochastic gradient algorithms have attracted much attention of researchers (cf., [24]- [29]). For example, George et al in [28] proposed a distributed stochastic gradient descent algorithm for solving non-convex optimization problems with applications in distributed super-vised learning.…”