Global Convergence and Stability of Stochastic Gradient Descent

Abstract

In machine learning, stochastic gradient descent (SGD) is widely deployed to train models using highly non-convex objectives with equally complex noise models. Unfortunately, SGD theory often makes restrictive assumptions that fail to capture the non-convexity of real problems, and almost entirely ignore the complex noise models that exist in practice. In this work, we make substantial progress on this shortcoming. First, we establish that SGD's iterates will either globally converge to a stationary point or diverge under nearly arbitrary nonconvexity and noise models. Under a slightly more restrictive assumption on the joint behavior of the non-convexity and noise model that generalizes current assumptions in the literature, we show that the objective function cannot diverge, even if the iterates diverge. As a consequence of our results, SGD can be applied to a greater range of stochastic optimization problems with confidence about its global convergence behavior and stability.

A Local Convergence Theory for the Stochastic Gradient Descent Method in Non-Convex Optimization With Non-isolated Local Minima

“…Our assumption is equivalent to the one in [11]). But it is weaker than those assumptions made in ( [4,34]).…”

“…There are several interesting remaining questions. First, whether SGD avoids saddle points has drawn much recent attention ( [13,20,23,31,34]). Studying this question under very mild assumptions would be interesting.…”

“…There has been exciting recent development for the analysis on SGD in non-convex optimization. [34] proved under the weak assumption that f has a locally Hölder continuous gradient, that the SGD iterations either converge to a stationary point or diverge almost surely with random initial guess. Another recent result is by [31], where they show, by assuming that the gradient is globally Lipchitz continuous and the set where the gradients with magnitude being less than some tolerance ǫ is bounded, the SGD iterations converge to stationary points almost surely with random initial guess.…”

“…For Theorems 3.1 and 3.2, we restate the results in [ [34], Appendix C] in order to suitably apply to our problems. Lemma 3.…”

“…A remarkable observation, as demonstrated in many recent studies, is that the stochastic gradient algorithms still perform well in non-convex settings, such as the deep neural networks ( [1,16,28]). This success has intrigued many theorists to explore convergence properties for non-convex optimization problems; almost sure convergence to critical points by [31,34], concentration inequalities for the difference between the loss function and its global minimum by [11], Convergence results using the Polyak-Lojasiewicz (PL) condition by [29,30] are some important examples of such results.…”