The Fastest Known Globally Convergent First-Order Method for Minimizing Strongly Convex Functions

Scoy, Bryan Van; Freeman, Randy A.; Lynch, Kevin

doi:10.1109/lcsys.2017.2722406

Cited by 114 publications

(91 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that the bound for the SSEP method was generated for the form (20); bounds for the efficient form (21) behave almost exactly like the bounds for the form (20)-the difference could not be observed from this plot-and were therefore omitted from the comparison. Finally, the numerical results presented above support the conjectures that the subspace-searching GFOM enjoys an O((1 − √ 2 √ κ −1 ) N R 2 x ) rate of convergence while the SSEP-based method that does not perform any line searches, converges at the faster rate O((1 − 2 √ κ −1 ) N R 2 x ) (matching the rate of convergence bounds on TMM [52,Theorem 1]). This is not in contraction with the theory, as noted in Remark 3, however, we are currently unable to find an intuitive explanation to this phenomenon besides the algebraic observation that the PEPs corresponding to the methods (20) and (21) have less degrees of freedom than the PEP for GFOM.…”

Section: Ssep-based Gradient Methods For Smooth Strongly Convex Minimisupporting

confidence: 72%

“…Since the approach is limited by our capability to accurately solve SDPs, it is important to note that PEPs can be used again for validating the performances of the final method (shown in the one before last row in Table 1). Figure 1 presents a comparison of the worst-case bounds in the case κ = 100 for GFOM, for an SSEP-based method performing no line searches, the celebrated fast (or accelerated) gradient method (FGM) for smooth strongly convex minimization [36,Theorem 2.1.12], and the very recent triple momentum method (TMM) [52]. These worst-case bounds were derived numerically by solving the corresponding PEPs using the interpolation conditions presented in Example 1 (see [13,50] for details on the derivation of PEPs for fixed-step methods).…”

Section: Ssep-based Gradient Methods For Smooth Strongly Convex Minimimentioning

confidence: 99%

“…Among others, such an approach was used in [13,22] to devise a fixed-step method that attains the best possible worst-case performance for smooth convex minimization [12], and later in [14] to obtain a variant of Kelley's cutting plane method with the best possible worst-case guarantee for non-smooth convex minimization. Also, the control-theoretic approach presented by Lessard et al in [26] was used in [52] for developing a new accelerated method for smooth strongly convex minimization, called the triple momentum method.…”

Section: Links With Systematic and Computer-assisted Approaches To Womentioning

confidence: 99%

See 2 more Smart Citations

Efficient first-order methods for convex minimization: a constructive approach

Drori

Taylor

2019

Math. Program.

View full text Add to dashboard Cite

We describe a novel constructive technique for devising efficient first-order methods for a wide range of large-scale convex minimization settings, including smooth, non-smooth, and strongly convex minimization. The technique builds upon a certain variant of the conjugate gradient method to construct a family of methods such that a) all methods in the family share the same worst-case guarantee as the base conjugate gradient method, and b) the family includes a fixed-step first-order method. We demonstrate the effectiveness of the approach by deriving optimal methods for the smooth and non-smooth cases, including new methods that forego knowledge of the problem parameters at the cost of a one-dimensional line search per iteration, and a universal method for the union of these classes that requires a three-dimensional search per iteration. In the strongly convex case, we show how numerical tools can be used to perform the construction, and show that the resulting method offers an improved worst-case bound compared to Nesterov's celebrated fast gradient method. IntroductionConvex optimization plays a central role in many fields of applications, including optimal control, machine learning and signal processing. In particular, when a large number of variables are involved within a convex optimization problem, the use of first-order methods is more and more widespread due to their typically very attractive low computational cost per iteration. This low computational cost comes, however, at a price: first-order methods often suffer from potentially slow convergence speeds, making them appropriate mostly for obtaining low to medium accuracy solutions. Nevertheless, first-order methods remain the methods of choice in many applications and currently receive a lot of attention from the optimization community, which constantly aims at improving them.An effective and fruitful approach used for analyzing and comparing first-order methods is the study of their worst-case behavior through the black-box model. In this setting, methods are only allowed to gain information on the objective through an oracle, which provides the value and the gradient of the objective at selected points.

show abstract

Section: Ssep-based Gradient Methods For Smooth Strongly Convex Minimisupporting

confidence: 72%

Section: Ssep-based Gradient Methods For Smooth Strongly Convex Minimimentioning

confidence: 99%

Section: Links With Systematic and Computer-assisted Approaches To Womentioning

confidence: 99%

See 1 more Smart Citation

Efficient first-order methods for convex minimization: a constructive approach

Drori

Taylor

2019

Math. Program.

View full text Add to dashboard Cite

show abstract

“…As z k = prox αg (2y k − x k ) and z = prox αg (2y − x ), we similarly conclude that e k Q (2) e k ≥ 0 is implied from the incremental QC that prox αg satisfies. Returning to (17), if we multiply from the left and right by e k and e k respectively, we obtain…”

Section: Convergence Certificates Via Matrix Inequalitiesmentioning

confidence: 99%

“…We now select algorithm parameters that satisfy the matrix inequality in (17). In doing so, we arrive at a new and simple proof of the O(1/k) convergence of DRS in the non-strongly convex and non-smooth case.…”

Section: ) Case 1: Non-strongly Convex and Non-smooth Casementioning

confidence: 99%

A Control-Theoretic Approach to Analysis and Parameter Selection of Douglas–Rachford Splitting

Seidman

Fazlyab

Preciado

et al. 2020

IEEE Control Syst. Lett.

View full text Add to dashboard Cite

Douglas-Rachford splitting and its equivalent dual formulation ADMM are widely used iterative methods in composite optimization problems arising in control and machine learning applications. The performance of these algorithms depends on the choice of step size parameters, for which the optimal values are known in some specific cases, and otherwise are set heuristically. We provide a new unified method of convergence analysis and parameter selection by interpreting the algorithm as a linear dynamical system with nonlinear feedback. This approach allows us to derive a dimensionally independent matrix inequality whose feasibility is sufficient for the algorithm to converge at a specified rate. By analyzing this inequality, we are able to give performance guarantees and parameter settings of the algorithm under a variety of assumptions regarding the convexity and smoothness of the objective function. In particular, our framework enables us to obtain a new and simple proof of the O(1/k) convergence rate of the algorithm when the objective function is not strongly convex.

show abstract

Non-monotone Behavior of the Heavy Ball Method

Danilova

Кулакова

Polyak

2020

Springer Proceedings in Mathematics &Amp; Statistics

View full text Add to dashboard Cite

We focus on the solutions of second-order stable linear difference equations and demonstrate that their behavior can be non-monotone and exhibit peak effects depending on initial conditions. The results are applied to the analysis of the accelerated unconstrained optimization method -the Heavy Ball method. We explain non-standard behavior of the method discovered in practical applications. In addition, such non-monotonicity complicates the correct choice of the parameters in optimization methods. We propose to overcome this difficulty by introducing new Lyapunov function which should decrease monotonically. By use of this function convergence of the method is established under less restrictive assumptions (for instance, with the lack of convexity). We also suggest some restart techniques to speed up the method's convergence.

show abstract

The Fastest Known Globally Convergent First-Order Method for Minimizing Strongly Convex Functions

Cited by 114 publications

References 10 publications

Efficient first-order methods for convex minimization: a constructive approach

Efficient first-order methods for convex minimization: a constructive approach

A Control-Theoretic Approach to Analysis and Parameter Selection of Douglas–Rachford Splitting

Non-monotone Behavior of the Heavy Ball Method

Contact Info

Product

Resources

About