An Explicit Convergence Rate for Nesterov's Method from SDP

We consider the problem of analyzing and designing gradient-based discrete-time optimization algorithms for a class of unconstrained optimization problems having strongly convex objective functions with Lipschitz continuous gradient. By formulating the problem as a robustness analysis problem and making use of a suitable adaptation of the theory of integral quadratic constraints, we establish a framework that allows to analyze convergence rates and robustness properties of existing algorithms and enables the design of novel robust optimization algorithms with prespecified guarantees capable of exploiting additional structure in the objective function.

show abstract

“…implies (33) for y ∈ p 2 . However, typically either only φ or the transformed operatorφ is bounded.…”

Section: Definition 6 (Doubly Hyperdominant Matrix) a Matrixmentioning

confidence: 98%

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

Michalowsky

Scherer

Ebenbauer

2020

International Journal of Control

View full text Add to dashboard Cite

show abstract

“…This also clarifies why various attempts to improve the rate by manual tuning [16] or sum-of-squares optimization [6] of the algorithm parameters were not successful. Our computation of an explicit optimal rate-bound for design is analogous to what has been achieved for the analysis of Nesterov's algorithm in [28]. Our approach brings out the intrinsic system theoretic reasons for the limits of performance in algorithm design; this holds for both the value of the optimal rate (determined by two zeros of some transfer matrix), and for the insight that algorithms (2.7) with matrices A of dimension larger than two are not beneficial.…”

Section: Convexification Of Operator Formulationmentioning

confidence: 97%

“…We then show in Section 4 how the special structure of the system can be exploited to convexify the common search for the algorithm parameters and the dynamic Zames-Falb multipliers which certify convergence. Our approach permits to derive explicit formulas for the optimal convergence rate that is achievable by synthesis, in analogy to the analysis results for Nesterov's algorithm in [28]. In this fashion, we are able to prove that the convergence rate of the triple momentum algorithm is indeed optimal if using the class of causal Zames-Falb multipliers to assure convergence.…”

mentioning

confidence: 95%

Convex Synthesis of Accelerated Gradient Algorithms

Scherer¹,

Ebenbauer²

2021

Preprint

View full text Add to dashboard Cite

We present a convex solution for the design of generalized accelerated gradient algorithms for strongly convex objective functions with Lipschitz continuous gradients. We utilize integral quadratic constraints and the Youla parameterization from robust control theory to formulate a solution of the algorithm design problem as a convex semidefinite program. We establish explicit formulas for the optimal convergence rates and extend the proposed synthesis solution to extremum control problems.

show abstract

“…Combining ( 27) with (28), it follows that V (x k , x k+1 )+(f k −f ) is a Lyapunov function that certifies geometric convergence with rate ρ. As with the case M m,L , it is possible to further augment the storage function and use more supply rates, which can potentially yield less conservative upper bounds on the worst-case convergence rate ρ.…”

Section: This Bound Can Be Minimized If η =mentioning

confidence: 99%

The Analysis of Optimization Algorithms, A Dissipativity Approach

Lessard

2022

Preprint

View full text Add to dashboard Cite

Optimization algorithms are ubiquitous across all branches of engineering, computer science, and applied mathematics. Typically, such algorithms are iterative in nature and seek to approximate the solution to a difficult optimization problem. The general form of an optimization problem is towhere x ∈ R d is the set of d real decision variables, C is the feasible set, and f : R d → R is the objective function. The goal is to find x ∈ C such that f (x) is as small as possible. For example, in structural mechanics, x could be lengths and widths of the members in a truss design, C could encode stress limitations of the materials and load bearing constraints, and f could be the total cost of the design. Solving this optimization problem would provide the cheapest truss design that satisfies all design specifications. In statistics, x could be parameters in a statistical model, C could correspond to constraints such as certain parameters being positive, and f could be the negative log-likelihood of the model given the observed data. Solving this optimization problem finds the maximum likelihood model. While some optimization problems can be solved analytically (such as least squares problems), analytical solutions do not exist for most optimization models, and numerical methods must be used to approximate the solution. Even when an analytical solution exists, it may be preferable to use numerical methods because they can be more computationally efficient. Iterative optimization algorithms typically begin with an estimate x 0 of the solution, and each step refines the estimate, producing a sequence x 0 , x 1 , . . . . If properly designed, then x k → x in the limit, and as many iterations as needed can be used to achieve the desired level of accuracy. Depending on the nature and structure of f and C in the optimization problem, different optimization algorithms may be appropriate. The design, selection, and tuning of optimization algorithms is more of an art than a science. Many different algorithms have been proposed, some with theoretical guarantees and others with a strong record of empirical performance. In practice, some algorithms converge slowly yet are predictable and reliable. Meanwhile, other algorithms converge more rapidly on average, but can fail spectacularly in some cases. Algorithm selection and tuning is typically performed by experts with deep area knowledge. In many ways, iterative algorithms behave like control systems, and the choice of algorithm is akin to the choice of controller. In the sections that follow, we formalize this connection and describe how optimization algorithms can be viewed as controllers performing robust control. This work will also show how dissipativity theory can be used to analyze the performance of many classes of optimization algorithms. This allows selection and tuning of optimization algorithms to be performed in an automated and systematic way.

show abstract

An Explicit Convergence Rate for Nesterov's Method from SDP

Cited by 5 publications

References 11 publications

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

Convex Synthesis of Accelerated Gradient Algorithms

The Analysis of Optimization Algorithms, A Dissipativity Approach

Contact Info

Product

Resources

About