Distributed dictionary learning

Abstract

The paper studies distributed Dictionary Learning (DL) problems where the learning task is distributed over a multi-agent network with time-varying (nonsymmetric) connectivity. This formulation is relevant, for instance, in Big Data scenarios where massive amounts of data are collected/stored in different spatial locations and it is unfeasible to aggregate and/or process all data in a fusion center, due to resource limitations, communication overhead or privacy considerations. We develop a general distributed algorithmic framework for the (nonconvex) DL problem and establish its asymptotic convergence. The new method hinges on Successive Convex Approximation (SCA) techniques coupled with i) a gradient tracking mechanism instrumental to locally estimate the missing global information; and ii) a consensus step, as a mechanism to distribute the computations among the agents. To the best of our knowledge, this is the first distributed algorithm with provable convergence for the DL problem and, more in general, bi-convex optimization problems over (time-varying) directed graphs

Improving the Sample and Communication Complexity for Decentralized Non-Convex Optimization: A Joint Gradient Estimation and Tracking Approach

“…Recent works also include the study on optimal convergence rates with respect to the network dependency for strongly convex [26] and convex [27] problems. When the problem becomes non-convex, many algorithms such as primal-dual based methods [28,10], gradient tracking based methods [11,29], and non-convex extensions of DGD methods [30] have been proposed, where the O(ǫ −1 ) iteration and communication complexity have been shown. Recently, optimal algorithm with respect to the network dependency has also been proposed in [13] with O(γ −1/2 × ǫ −1 ) computation and O(ǫ −1 ) communication complexity, where γ denotes the spectral gap of the communication graph G. Note that the above algorithms all require O(1) full gradient evaluations per iteration, so when directly applied to solve problems where each f i (•) takes the form in (2), they all require O(mnǫ −1 ) local data samples.…”

Distributed Non-Convex First-Order optimization and Information Processing: Lower Complexity Bounds and Rate Optimal Algorithms

“…References [14,15] propose a primal-dual based method for unconstrained problem over a connected network, and derives a global convergence rate for this setting. In [13,17,18], the authors utilize certain gradient tracking idea to solve a constrained nonsmooth distributed problem over possibly time-varying networks. The work [19] summarizes a number of recent progress in extending the DSG-based methods for non-convex problems.…”

“…The problem (1) and (2) have been studied extensively in the literature when f i 's are all convex; see for example [4][5][6]. Primal based methods such as distributed subgradient (DSG) method [4], the EXTRA method [6], as well as primal-dual based methods such as distributed augmented Lagrangian method [7], Alternating Direction Method of Multipliers (ADMM) [8,9] have been proposed.On the contrary, only recently there have been works addressing the more challenging problems without assuming convexity of f i ; see [1,3,[10][11][12][13][14][15][16][17][18][19][20][21][22][23]. The convergence behavior of the distributed consensus problem (1) has been studied in [3,10,11].…”

“…On the contrary, only recently there have been works addressing the more challenging problems without assuming convexity of f i ; see [1,3,[10][11][12][13][14][15][16][17][18][19][20][21][22][23]. The convergence behavior of the distributed consensus problem (1) has been studied in [3,10,11].…”

Decentralized Dictionary Learning Over Time-Varying Digraphs

“…Recently and independently from our conference work (Daneshmand et al, 2016), Zhao et al (2016) proposed a distributed primal-dual-based method for a class of dictionary learning problems related, but different from Problem P. Specifically, they considered: quadratic loss functions f i , with a quadratic regularization on the dictionary (i.e., G = 0), and norm ball constraints on the private variables. The network is modeled as a fixed undirected graph.…”