Distributed Training for Multi-Layer Neural Networks by Consensus

Abstract

Over the past decade, there has been a growing interest in large-scale and privacy-concerned machine learning, especially in the situation where the data cannot be shared due to privacy protection or cannot be centralized due to computational limitations. Parallel computation has been proposed to circumvent these limitations, usually based on the master-slave and decentralized topologies, and the comparison study shows that a decentralized graph could avoid the possible communication jam on the central agent but incur extra communication cost. In this brief, a consensus algorithm is designed to allow all agents over the decentralized graph to converge to each other, and the distributed neural networks with enough consensus steps could have nearly the same performance as the centralized training model. Through the analysis of convergence, it is proved that all agents over an undirected graph could converge to the same optimal model even with only a single consensus step, and this can significantly reduce the communication cost. Simulation studies demonstrate that the proposed distributed training algorithm for multi-layer neural networks without data exchange could exhibit comparable or even better performance than the centralized training model.

Consensus-Based Cooperative Algorithms for Training Over Distributed Data Sets Using Stochastic Gradients

“…O PTIMISATION and learning over distributed networks have been widely studied in recent years, owing to their significant potentials in many biological, engineering, and social applications [1][2][3][4][5][6]. Several critical limitations of the centralised methods can be addressed by the distributed algorithms: first, communicational requirement is relieved as information exchanges are confined to adjacent neighbours; second, local datasets can be kept private and do not need to be revealed to remote fusion centres; third, computational burdens are distributed into a set of agents, where each of them only needs to process its local datasets.…”

“…More recently, considerable interests have been concentrated on training neural networks using distributed datasets [4,[13][14][15]. One of the most important reasons is that we are now faced with the era of big data, and massive amounts of information are generated everyday and everywhere.…”

“…To deal with those challenges, significant efforts have been dedicated to distributed learning [4,13,17,18]. Various approaches have been proposed in the existing literature, for example, the incremental strategies [19][20][21], and the consensusbased strategies [4,6,13].…”

Differential Graphical Games for Constrained Autonomous Vehicles Based on Viability Theory

“…The distributed control problem of large complex network systems is always one of the major barriers in industrial applications. Various studies on distributed control have pointed to air traffic control, sensor networks, satellite formation control, multi-robot docking, distributed learning, and so on [1], [2]. The aim of distributed control is to design a control policy for each connected agent such that the evolutions of all the agents will finally contribute to the desired behaviour of the network system [3].…”

Distributed Heuristic Adaptive Neural Networks With Variance Reduction in Switching Graphs

“…However, each agent needs to communicate with its neighbors many times to achieve consensus, which is computationally expensive. Liu et al [21] proposed a distributed neural networks framework based on the consensus algorithm, where all agents only need a single consensus step with their connected neighbors after every training update. Lian et al [22] theoretically analyzed the centralized and decentralized algorithms on their convergence rate, which shows that the decentralized method has a faster convergence rate than its centralized counterpart on high latency or low bandwidth system and can bring asymptotically linear speed up when more agents are available.…”