Asynchronous variance-reduced block schemes for composite non-convex stochastic optimization: block-specific steplengths and adapted batch-sizes

Abstract

We consider the minimization of a sum of an expectation-valued coordinate-wise L i -smooth nonconvex function and a nonsmooth block-separable convex regularizer. Prior schemes are characterized by the following shortcomings: (a) Steplengths require global knowledge of Lipschitz constants; (b) Batchsizes of gradients are centrally updated and require knowledge of the global clock; (c) a.s. convergence guarantees are unavailable; (d) Rates are inferior compared to deterministic counterparts. Specifically, (a) and (b) require coordination across blocks and necessitate global information, leading to potentially larger constants in the rate and the oracle complexity bounds, impeding decentralized implementations, and resulting in relatively poor empirical behavior. We address these shortcomings by proposing an asynchronous variance-reduced algorithm, where in each iteration, a single block is randomly chosen to update its estimates by a proximal variable sample-size stochastic gradient scheme, while the remaining blocks are kept invariant. Notably, each block employs a steplength that is in accordance with its blockspecific Lipschitz constant while block-specific batch-sizes are random variables updated at a rate that grows either at a geometric or polynomial rate with the (random) number of times that block is selected. We show that every limit point for almost every sample path is a stationary point and establish the ergodic non-asymptotic rate O(1/K). Iteration and oracle complexity to obtain an -stationary point are shown to be O(1/ ) and O(1/ 2 ), respectively. Furthermore, under a µ-proximal Polyak-Łojasiewicz (PL) condition with the batch size increasing at a geometric rate, we prove that the suboptimality diminishes at a geometric rate, the optimal deterministic rate while iteration and oracle complexity to obtain an -optimal solution are proven to be O((L max /µ) ln(1/ )) and O (L ave /µ)(1/ ) 1+c with c ≥ 0, respectively. In the single block setting, we obtain the optimal oracle complexity bound O(1/ ). In pursuit of less aggressive sampling rates, when the batch sizes increase at a polynomial rate of degree v ≥ 1, suboptimality decays at a corresponding polynomial rate while the iteration and oracle complexity to obtain an −optimal solution are provably O(v(1/ ) 1/v ) and O e v v 2v+1 (1/ ) 1+1/v , respectively. Finally, preliminary numerics support our theoretical findings, displaying significant improvements over schemes where steplengths are based on global Lipschitz constants. * The authors are with the

Variance-Reduced Accelerated First-order Methods: Central Limit Theorems and Confidence Statements

“…Based on Lemmas 1, 2, and 3, we can establish the geometric rate of convergence along with the iteration and oracle complexity of the iterates generated by Algorithms 1, 2, and 3. The proofs of Propositions 1, 2, and 3 are similar to that of [26,Theorem 4.2 and Corollary 4.7]. Related results for linear convergence for stochastic gradient methods can be found in [8,15,23,36,43] while a linear rate for the accelerated variants has been provided in [22].…”

“…Unfortunately, SA schemes with diminishing steps cannot recover the deterministic convergence rates seen in exact gradient methods while constant steplength SA schemes are only characterized by convergence guarantees to a neighborhood of the optimal solution. Variance-reduction schemes employing an increasing batch-size of sampled gradients (instead of the unavailable true gradient) appear to have been first alluded to in [1,12,20,35] and analyzed in smooth and strongly convex [8,15,42,43], smooth convex regimes [17], nonsmooth (but smoothable) convex [22], nonconvex [26], and game-theoretic [27] regimes. Notably, the linear rates in mean-squared error were derived in for strongly convex smooth [43] and a subclass of nonsmooth objectives [22], while a rate O(1/k 2 ) and O(1/k) was obtained for expected sub-optimality in convex smooth [17,23] and nonsmooth [22], respectively.…”

Zeroth-order randomized block methods for constrained minimization of expectation-valued Lipschitz continuous functions

“…& Management, Oklahoma State University, farzad.yousefian@okstate.edu; Yousefian acknowledges the support of the NSF through CAREER grant ECCS-1944500. nonsmoothness arises in a deterministic form . However, in many applications, f (•, ω) may be both nonconvex and nonsmooth and proximal stochastic gradient schemes [10], [14] cannot be directly adopted. We now discuss some relevant research in nonsmooth and nonconvex regimes.…”

“…In [2], the authors prove that a limit point of a subsequence is an ǫ-Clarke stationary point when f is locally Lipschitz while Kiwiel proved that every limit point is Clarke stationary with respect to f without requiring compactness of level sets [13]. There have also been efforts to develop statements in structured regimes where f is either weakly convex [7], [8] or f = g+h and h is smooth and possibly nonconvex while g is convex, nonsmooth, and proximable [14], [21].…”

“…Much of the efforts in the regime of stochastic nonconvex optimization have been restricted to structured regimes where f (x) = h(x) + g(x), h(x) E[F (x, ω)], h is smooth and possibly nonconvex while g is closed, convex, and proper with an efficient proximal evaluation. In such settings, proximal stochastic gradient techniques [10] and their variancereduced counterparts [10], [14] were developed.…”