Distributed Heuristic Adaptive Neural Networks With Variance Reduction in Switching Graphs

Abstract

This article proposes a distributed adaptive training method for neural networks in switching communication graphs to deal with the problems concerned with massive data or privacy-related data. First, the stochastic variance reduced gradient (SVRG) is used for the training of neural networks. Then, the authors propose a heuristic adaptive consensus algorithm for distributed training, which adaptively adjusts the weighted connectivity matrix based on the performance of each agent over the communication graph. Furthermore, it is proved that the proposed distributed heuristic adaptive neural networks ensure the convergence of all the agents to the optimum with a single communication among connected neighbors after every training step, which is also suitable for switching graphs. This theorem is verified by the simulation, which gives the results that fewer iterations are required for all agents to reach the optimum using the proposed heuristic adaptive consensus algorithm, and the SVRG can greatly decrease the fluctuations caused by the stochastic gradient and improve its performance with only a little extra computational cost.

On the game‐theoretic analysis of distributed generative adversarial networks

“…Distributed training for multiple GANs over networks has not been well‐examined in the existing literature, and this motivates us to develop a framework based on the recent advances in distributed Nash equilibrium seeking by formulating a zero‐sum game between two adversarial networks of discriminators and generators. It is distinct from the extant results on distributed learning as in References [12–14,18,29,30], where only cooperation among agents exists, while learning for GANs inherently shows its nature of competition. In other words, the existing works 12–14,18 are intrinsically distributed optimisation problems (see References [31–35]) whereas training for GANs in this study is a distributed Nash equilibrium seeking problem with two coalitions (see References [21,23,28,36,37]).…”

Consensus-Based Cooperative Algorithms for Training Over Distributed Data Sets Using Stochastic Gradients

“…Therefore, we will adopt the full distributed learning networks without any central node to avoid possible failures and communication latency. Detailed advantages and comparisons of the two types of networks have been well documented in recent studies [4,6].…”

“…O PTIMISATION and learning over distributed networks have been widely studied in recent years, owing to their significant potentials in many biological, engineering, and social applications [1][2][3][4][5][6]. Several critical limitations of the centralised methods can be addressed by the distributed algorithms: first, communicational requirement is relieved as information exchanges are confined to adjacent neighbours; second, local datasets can be kept private and do not need to be revealed to remote fusion centres; third, computational burdens are distributed into a set of agents, where each of them only needs to process its local datasets.…”

“…To deal with those challenges, significant efforts have been dedicated to distributed learning [4,13,17,18]. Various approaches have been proposed in the existing literature, for example, the incremental strategies [19][20][21], and the consensusbased strategies [4,6,13]. In the incremental strategies, a cyclic path that traverses all the agents in the network is required at every iteration, which is computationally intensive and also sensitive to link failures.…”

“…Distributed support vector machine (SVM) learning for wireless sensor networks is considered in [17]. Stochastic variance reduction methods are developed in [6] by using batch gradient descent algorithms. In general, strong assumptions, e.g., convergent learning parameters and basin of attraction, have been utilised to establish the convergence of those algorithms in the existing studies.…”