Stopping Criteria for, and Strong Convergence of, Stochastic Gradient Descent on Bottou-Curtis-Nocedal Functions

Abstract

While Stochastic Gradient Descent (SGD) is a rather efficient algorithm for data-driven problems, it is an incomplete optimization algorithm as it lacks stopping criteria, which has limited its adoption in situations where such criteria are necessary. Unlike stopping criteria for deterministic methods, stopping criteria for SGD require a detailed understanding of (A) strong convergence, (B) whether the criteria will be triggered, (C) how false negatives are controlled, and (D) how false positives are controlled. In order to address these issues, we first prove strong global convergence (i.e., convergence with probability one) of SGD on a popular and general class of convex and nonconvex functions that are specified by, what we call, the Bottou-Curtis-Nocedal structure. Our proof of strong global convergence refines many techniques currently in the literature and employs new ones that are of independent interest. With strong convergence established, we then present several stopping criteria and rigorously explore whether they will be triggered in finite time and supply bounds on false negative probabilities. Ultimately, we lay a foundation for rigorously developing stopping criteria for SGD methods for a broad class of functions, in hopes of making SGD a more complete optimization algorithm with greater adoption for data-driven problems.

Global Convergence and Stability of Stochastic Gradient Descent

“…Importantly, {M k } cannot be arbitrary, and the following properties specify a generalization of the Robbins & Monro (1951) conditions for matrix-valued learning rates (c.f. Patel, 2020).…”

“…To show convergence to zero, however, is not standard. Several strategies have been developed, namely those of Li & Orabona (2019); ; Mertikopoulos et al (2020); Patel (2020). Unfortunately, the approaches of Li & Orabona (2019); rely intimately on the existence of a global Hölder constant, while that of Mertikopoulos et al (2020) requires even more restrictive assumptions.…”

“…Accordingly, understanding SGD's behavior has been crucial to its reliable application in machine learning and beyond. As a result, SGD's theory has greatly advanced from a variety of perspectives: global convergence analysis Gower et al, 2020;Khaled & Richtárik, 2020;Mertikopoulos et al, 2020;Patel, 2020), local convergence analysis (Mertikopoulos et al, 2020), greedy and global complexity analysis (Gower et al, 2020;Khaled & Richtárik, 2020), asymptotic weak convergence (Wang et al, 2021), and saddle point analysis (Fang et al, 2019;Mertikopoulos et al, 2020;Jin et al, 2021). While all of these perspectives add new dimensions to our understanding of SGD, the global convergence analysis of SGD is the foundation as it dictates whether local analyses, complexity analyses or saddle point analyses are even warranted.…”

“…1. a global Hölder constant for the gradient of the objective function (e.g., Reddi et al, 2016a;Ma & Klabjan, 2017;Zhou et al, 2018;Bassily et al, 2018;Li & Orabona, 2019;Gower et al, 2020;Khaled & Richtárik, 2020;Mertikopoulos et al, 2020;Patel, 2020;Jin et al, 2021); 2. unrealistic noise models (e.g., uniformly bounded variance) for the stochastic gradients (e.g., Reddi et al, 2016b;Ma & Klabjan, 2017;Bi & Gunn, 2019;Zou et al, 2019;Mertikopoulos et al, 2020).…”

“…Specifically, our results allow for SGD's iterates to converge to stationary point or diverge; as a result, when SGD's iterates converge to a stationary point, saddle point escape analyses (e.g., Fang et al, 2019;Mertikopoulos et al, 2020;Jin et al, 2021) can be applied to ensure that the stationary point is a local minimizer. Moreover, when SGD is converging to a stationary point, local convergence rate analyses can also be supplied, which can inform adaptive step size rules and stopping criteria (Patel, 2020).…”