A Newton-CG Algorithm with Complexity Guarantees for Smooth Unconstrained Optimization

Global Optimality in Distributed Low-rank Matrix Factorization

“…From Theorem 6, we then conclude that conditioned on observing that {z(k)} ⊆ B ρ and all limit points of {z(k)} are in B ρ , DGD+LOCAL converges to a critical point of the objective function in (19), and the probability that this critical point is a strict saddle point is zero. We refer to this point as z ⋆ .…”

“…Proposition 3 thus guarantees that (19) has no critical points outside of the consensus subspace. Since we have argued that DGD+LOCAL converges to a second-order critical point z ⋆ of ( 19), it follows that z ⋆ must be on the consensus subspace; that is, (16), which is exactly equivalent to problem (14).…”

“…Proof. We begin by arguing that DGD+LOCAL converges almost surely (when z(0) is chosen randomly inside B ρ ) to a second-order critical point of (19). To do this, our goal is to invoke Theorem 6.…”

“…Theorem 8 tells us that problem (14) has two types of critical points: global minimizers and strict saddles. If (U ⋆ , V ⋆ ) were a strict saddle point of ( 14), Theorem 7 tells us that z ⋆ must then be a strict saddle of (19). However, z ⋆ is almost surely a second-order critical point of (19), where the Hessian has no negative eigenvalues.…”

“…In parallel with the recent focus on the favorable geometry of certain nonconvex landscapes, it has been shown that a number of local search algorithms have the capability to avoid strict saddle points and converge to a local minimizer for problems that satisfy the strict saddle property [10,12,13,19]. As stated in [12] and as we summarize in Theorems 2 and 4, gradient descent when started from a random initialization is one such algorithm.…”