eg-VSSA: An extragradient variable sample-size stochastic approximation scheme: Error analysis and complexity trade-offs

Jalilzadeh, Afrooz; Uday, V Shanbhag

doi:10.1109/wsc.2016.7822133

Cited by 7 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then for η < (4(n + 1) 2 /τ 2 ) 1/3 , we have d < 1. Furthermore, by recalling that N k = N 0 q −k , it follows that d k ≤ 5ην 2 1 q k 16N 0 , we obtain the following bound from (16).…”

Section: Nonsmooth Strongly Convex Optimizationmentioning

confidence: 76%

“…By utilizing variable sample-size (VS) stochastic gradient schemes, linear convergence rates were obtained for strongly convex problems [36,18] and these rates were subsequently improved (in a constant factor sense) through a VS-accelerated proximal method developed by Jalilzadeh et al [17] (called (VS-APM)). In convex regimes, Ghadimi and Lan [12] developed an accelerated framework that admits the optimal rate of O(1/k 2 ) and the optimal oracle complexity (also see [18]), improving the rate statement presented in [16]. More recently, in [17], Jalilzadeh et al present a smoothed accelerated scheme that admits the optimal rate of O(1/k) and optimal oracle complexity for nonsmooth problems, recovering the findings in [12] in the smooth regime.…”

Section: Convexitymentioning

confidence: 99%

“…For instance, in standard stochastic gradient schemes for strongly convex smooth problems with Lipschitz continuous gradients, the mean-squared error diminishes at a rate of O(1/k) while deterministic schemes display a geometric rate of convergence. This gap can be reduced by utilizing an increasing sample-size of gradients, an approach first considered in [11,6], and subsequently refined for gradient-based methods for strongly convex [36,18,17], convex [16,12,18,17], and nonsmooth convex regimes [17]. Variance-reduced techniques have also been considered for stochastic quasi-Newton (SQN) techniques [23,45,5] under twice differentiability and strong convexity requirements.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Variable Sample-Size Stochastic Quasi-Newton Method for Smooth and Nonsmooth Stochastic Convex Optimization

Jalilzadeh

Nedić

Shanbhag

et al. 2018

2018 IEEE Conference on Decision and Control (CDC)

Self Cite

View full text Add to dashboard Cite

Classical theory for quasi-Newton schemes has focused on smooth deterministic unconstrained optimization while recent forays into stochastic convex optimization have largely resided in smooth, unconstrained, and strongly convex regimes. Naturally, there is a compelling need to address nonsmoothness, the lack of strong convexity, and the presence of constraints. Accordingly, this paper presents a quasi-Newton framework that can process merely convex and possibly nonsmooth (but smoothable) stochastic convex problems. We propose a framework that combines iterative smoothing and regularization with a variance-reduced scheme reliant on using an increasing sample-size of gradients. We make the following contributions. (i) We develop a regularized and smoothed variable sample-size BFGS update (rsL-BFGS) that generates a sequence of Hessian approximations and can accommodate nonsmooth convex objectives by utilizing iterative regularization and smoothing. (ii) In strongly convex regimes with statedependent noise, the proposed variable sample-size stochastic quasi-Newton (VS-SQN) scheme admits a non-asymptotic linear rate of convergence while the oracle complexity of computing an -solution is O(κ m+1 / ) where κ denotes the condition number and m ≥ 1. In nonsmooth (but smoothable) regimes, using Moreau smoothing retains the linear convergence rate while using more general smoothing leads to a deterioration of the rate to O(k −1/3 ) for the resulting smoothed VS-SQN (or sVS-SQN) scheme. Notably, the nonsmooth regime allows for accommodating convex constraints; (iii) In merely convex but smooth settings, the regularized VS-SQN scheme rVS-SQN displays a rate of O(1/k (1−ε) ) with an oracle complexity of O(1/ 3 ). When the smoothness requirements are weakened, the rate for the regularized and smoothed VS-SQN scheme rsVS-SQN worsens to O(k −1/3 ). Such statements allow for a state-dependent noise assumption under a quadratic growth property on the objective. To the best of our knowledge, the rate results are amongst the first available rates in nonsmooth regimes. Preliminary numerical evidence suggests that the schemes compare well with accelerated gradient counterparts on selected problems in stochastic optimization and machine learning with significant benefits in ill-conditioned regimes.

show abstract

Section: Nonsmooth Strongly Convex Optimizationmentioning

confidence: 76%

Section: Convexitymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Variable Sample-Size Stochastic Quasi-Newton Method for Smooth and Nonsmooth Stochastic Convex Optimization

Jalilzadeh

Nedić

Shanbhag

et al. 2018

2018 IEEE Conference on Decision and Control (CDC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Randomized smoothing techniques have also been employed by [38] together with recursive steplengths (see [25] for a review) (b) Variance reduction. In strongly convex regimes (without acceleration), a linear rate of convergence in expected error was first shown for variance-reduced gradient methods by [33] and revisited by [15], while similar rates were provided for extragradient methods by [14]; the accelerated counterpart (VS-APM) improves the complexity to O( L/µ log(1/ )). In smooth regimes, an accelerated scheme was first presented by [12] where every iteration requires two prox evals., admitting the optimal rate and oracle complexity of O(1/k 2 ) and O(1/ 2 ), respectively.…”

Section: Prior Researchmentioning

confidence: 83%

Smoothed Variable Sample-size Accelerated Proximal Methods for Nonsmooth Stochastic Convex Programs

Jalilzadeh¹,

Shanbhag²,

Blanchet³

2018

Preprint

Self Cite

View full text Add to dashboard Cite

We consider minimizing f (x) = E[f (x, ω)] when f (x, ω) is possibly nonsmooth and either strongly convex or convex in x. (I) Strongly convex. When f (x, ω) is µ−strongly convex in x, traditional stochastic approximation (SA) schemes often display poor behavior, arising in part from noisy subgradients and diminishing steplengths. Instead, we propose a variable sample-size accelerated proximal scheme (VS-APM) and apply it on f η (x), the (η-)Moreau smoothed variant of E[f (x, ω)]; we term such a scheme as (η-VS-APM). In contrast with SA schemes, (η-VS-APM) utilizes constant steplengths and increasingly exact gradients, achieving an optimal oracle complexity in stochastic subgradients of O(1/ ) with an iteration complexity of O( (ηµ + 1)/(ηµ) log(1/ )) in inexact (outer) gradients of f η (x), computed via an increasing number of inner stochastic subgradient steps. This approach is also beneficial for ill-conditioned L-smooth problems where L/µ is massive, resulting in better conditioned outer problems and allowing for larger steps and better numerical behavior. This framework is characterized by an optimal oracle complexity of O( L/µ + 1/(ηµ) log(1/ )) and an overall iteration complexity of O(log 2 (1/ )) in gradient steps. (II) Convex. When f (x, ω) is merely convex but smoothable, by suitable choices of the smoothing, steplength, and batch-size sequences, smoothed (VS-APM) (or sVS-APM) produces sequences for which expected sub-optimality diminishes at the rate of O(1/k) with an optimal oracle complexity of O(1/ 2 ). Our results can be specialized to two important cases: (a) Smooth f . Since smoothing is no longer required, we observe that (VS-APM) admits the optimal rate and oracle complexity, matching prior findings; (b) Deterministic nonsmooth f . In the nonsmooth deterministic regime, (sVS-APM) reduces to a smoothed accelerated proximal method (s-APM) that is both asymptotically convergent, admitting a non-asymptotic rate of O(1/k), matching that by [23] for producing approximate solutions. Finally, (sVS-APM) and (VS-APM) produce sequences that converge almost surely to a solution of the original problem.

show abstract

“…Over the last 15 years, there has been a surge of advances in stochastic first-order schemes, including extragradient methods (Juditsky et al 2011, Yousefian et al 2014, Jalilzadeh and Shanbhag 2016, accelerated schemes Lan 2016, Jalilzadeh et al 2018b), variance-reduced schemes (Shanbhag and Blanchet 2015, Jofré and Thompson 2019), deterministic step length schemes for nonconvex programs , and SQN schemes (Lucchi et al 2015, Zhou et al 2017, Jalilzadeh et al 2018a. We review some recent advances in stochastic trust-region and line-search methods.…”

Section: Stochastic Iterative Schemes For Stochastic Nonconvex Optimimentioning

confidence: 99%

Open Problem—Iterative Schemes for Stochastic Optimization: Convergence Statements and Limit Theorems

2019

Self Cite

View full text Add to dashboard Cite

Central limit theorems represent among the most celebrated of limit theorems in probability theory ( Lindeberg 1922 , Feller 1945 ). It may be recalled that the sum of n independent and identically distributed zero mean square integrable random variables grows at the rate of [Formula: see text]. Consequently, by dividing this sum by [Formula: see text], we expect its law to possibly approach a limit. This intuition is formalized by the central limit theorem (CLT; Chow and Teicher 2012 , chapter 9), and in fact, this limit is the normal distribution. Our interest lies in developing limiting statements for estimators to the following stochastic optimization problem:[Formula: see text]where f(x) is a strongly convex function on [Formula: see text], F : [Formula: see text] × [Formula: see text] → [Formula: see text] is a real-valued function, [Formula: see text] : Ω → [Formula: see text], and (Ω,[Formula: see text],[Formula: see text]) denotes the associated probability space.

show abstract

eg-VSSA: An extragradient variable sample-size stochastic approximation scheme: Error analysis and complexity trade-offs

Cited by 7 publications

References 12 publications

A Variable Sample-Size Stochastic Quasi-Newton Method for Smooth and Nonsmooth Stochastic Convex Optimization

A Variable Sample-Size Stochastic Quasi-Newton Method for Smooth and Nonsmooth Stochastic Convex Optimization

Smoothed Variable Sample-size Accelerated Proximal Methods for Nonsmooth Stochastic Convex Programs

Open Problem—Iterative Schemes for Stochastic Optimization: Convergence Statements and Limit Theorems

Contact Info

Product

Resources

About