A consensus-based decentralized algorithm for non-convex optimization with application to dictionary learning

Abstract

In handling massive-scale signal processing problems arising from 'big-data' applications, key technologies could come from the development of decentralized algorithms. In this context, consensusbased methods have been advocated because of their simplicity, fault tolerance and versatility. This paper presents a new consensus-based decentralized algorithm for a class of non-convex optimization problems that arises often in inference and learning problems, including 'sparse dictionary learning' as a special case. For the proposed algorithm, we provide sufficient conditions for convergence to a stationary point. Numerical results demonstrate the efficacy of the proposed algorithm and provide evidence that validates our convergence claim.

Distributed dictionary learning

“…Our results can be contrasted with [17]- [21] wherein gradient schemes tailored with consensus/diffusion updates are employed for some instance of (P1). The aforementioned schemes do not have any convergence guarantees: it is postulated that the sequence generated by the algorithms is convergent (see, e.g., [20], [21]), and then concluded that any limit point is a stationary solution of the problem. Furthermore, some of these schemes do not even achieve consensus among the local variables.…”

Distributed Non-Convex First-Order optimization and Information Processing: Lower Complexity Bounds and Rate Optimal Algorithms

“…It follows that using an algorithm in A or A , it takes at least M/3 iterations for the non-zero x r [2] and the corresponding gradient vector to propagate to at least one node in set [2M/3 + 1, M ]. Once we have x j [2] = 0 for some j ∈ [2M/3 + 1, M ], then according to (57), it is possible to have ∂h j (x j ) ∂x j [3] = 0, and once this gradient becomes non-zero, the corresponding variable…”

“…The problem (1) and (2) have been studied extensively in the literature when f i 's are all convex; see for example [4][5][6]. Primal based methods such as distributed subgradient (DSG) method [4], the EXTRA method [6], as well as primal-dual based methods such as distributed augmented Lagrangian method [7], Alternating Direction Method of Multipliers (ADMM) [8,9] have been proposed.On the contrary, only recently there have been works addressing the more challenging problems without assuming convexity of f i ; see [1,3,[10][11][12][13][14][15][16][17][18][19][20][21][22][23]. The convergence behavior of the distributed consensus problem (1) has been studied in [3,10,11].…”

“…On the contrary, only recently there have been works addressing the more challenging problems without assuming convexity of f i ; see [1,3,[10][11][12][13][14][15][16][17][18][19][20][21][22][23]. The convergence behavior of the distributed consensus problem (1) has been studied in [3,10,11]. Reference [12] develops a non-convex ADMM based methods for solving the distributed consensus problem (1).…”

Distributed dictionary learning via projections onto convex sets

“…In the second one (and also in [12] where a more extended version can be found), a distributed version of the centralized K-SVD algorithm is proposed, while convergence analysis of the algorithm is presented in [13]. Additionally, in [14], a consensus-based, distributed algorithm for general inference / learning problems is proposed which can also be applied for the problem of dictionary learning. An online algorithm has appeared in [15], where the recursive least squares algorithm is employed.…”