Global Optimality in Distributed Low-rank Matrix Factorization

Abstract

We study the convergence of a variant of distributed gradient descent (DGD) on a distributed lowrank matrix approximation problem wherein some optimization variables are used for consensus (as in classical DGD) and some optimization variables appear only locally at a single node in the network. We term the resulting algorithm DGD+LOCAL. Using algorithmic connections to gradient descent and geometric connections to the well-behaved landscape of the centralized low-rank matrix approximation problem, we identify sufficient conditions where DGD+LOCAL is guaranteed to converge with exact consensus to a global minimizer of the original centralized problem. For the distributed low-rank matrix approximation problem, these guarantees are stronger-in terms of consensus and optimality-than what appear in the literature for classical DGD and more general problems.

Provable Bregman-divergence based Methods for Nonconvex and Non-Lipschitz Problems

“…Finally, as many popular (nonconvex) machine learning and signal processing problems [7,9,14,15,20,[22][23][24] have such a landscape property as all second-order stationary points are globally optimal solutions, the global optimality can be achieved by the proposed Bregmandivergence based algorithms in solving this particular class of problems.…”

The Landscape of Non-convex Empirical Risk with Degenerate Population Risk

“…The landscape of the above population risk has been studied in the general R N ×k space with k = r in [7]. The landscape of its variants, such as the asymmetric version with or without a balanced term, has also been studied in [4,18]. It is well known that there exists an ambiguity in the solution of (3.2) due to the fact that UU = UQQ U holds for any orthogonal matrix Q ∈ R k×k .…”