Exponential convergence bounds using integral quadratic constraints

Boczar, Ross; Lessard, Laurent; Recht, Benjamin

doi:10.1109/cdc.2015.7403406

Cited by 19 publications

(40 citation statements)

References 30 publications

(81 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Theorem 4 is a generalization of the IQCs presented in [5], [17] also allowing for anticausal multipliers Π, i.e., anticausal transfer matrices E in (41). The class of multipliers from [5], [17] is then obtained by letting + = 0. We note that an extension to anticausal multipliers has also been proposed in [15] using a slightly different approach.…”

Section: Definition 6 (Doubly Hyperdominant Matrix) a Matrixmentioning

confidence: 99%

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

Michalowsky

Scherer

Ebenbauer

2020

International Journal of Control

View full text Add to dashboard Cite

We consider the problem of analyzing and designing gradient-based discrete-time optimization algorithms for a class of unconstrained optimization problems having strongly convex objective functions with Lipschitz continuous gradient. By formulating the problem as a robustness analysis problem and making use of a suitable adaptation of the theory of integral quadratic constraints, we establish a framework that allows to analyze convergence rates and robustness properties of existing algorithms and enables the design of novel robust optimization algorithms with prespecified guarantees capable of exploiting additional structure in the objective function.

show abstract

Section: Definition 6 (Doubly Hyperdominant Matrix) a Matrixmentioning

confidence: 99%

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

Michalowsky

Scherer

Ebenbauer

2020

International Journal of Control

View full text Add to dashboard Cite

show abstract

“…The nonlinearity of interest is sector-bounded and slope-restricted because it is the gradient of a function g ∈ F(0, κ − 1). We may therefore represent the nonlinearity with a Zames-Falb IQC as in [13], leading to…”

Section: Control Design Interpretationsmentioning

confidence: 99%

“…In Figure 2 (left panel), we show the Nyquist plot for the Gradient Method using the sector IQC [3,13]. To this effect, we set β = γ = 0 and use either α = 2 L+m or α = 1 L .…”

Section: Control Design Interpretationsmentioning

confidence: 99%

A Robust Accelerated Optimization Algorithm for Strongly Convex Functions

Cyrus

Scoy

et al. 2018

2018 Annual American Control Conference (ACC)

Self Cite

View full text Add to dashboard Cite

This work proposes an accelerated first-order algorithm we call the Robust Momentum Method for optimizing smooth strongly convex functions. The algorithm has a single scalar parameter that can be tuned to trade off robustness to gradient noise versus worst-case convergence rate. At one extreme, the algorithm is faster than Nesterov's Fast Gradient Method by a constant factor but more fragile to noise. At the other extreme, the algorithm reduces to the Gradient Method and is very robust to noise. The algorithm design technique is inspired by methods from classical control theory and the resulting algorithm has a simple analytical form. Algorithm performance is verified on a series of numerical simulations in both noise-free and relative gradient noise cases.Notation. The set of functions that are m-strongly convex and L-smooth is denoted F(m, L). In particular, f ∈ F(m, L) if for all x, y ∈ R n ,The condition ratio is defined as κ := L/m.2 A numerical study in [3] revealed that the standard rate bound for FGM derived in [2] is conservative. Nevertheless, the bound has a simple algebraic form and is asymptotically tight.

show abstract

“…For any optimization method F u (P, K) described by (11), one can form an associated interconnection [P, K] by adding the signals (r, e) and fixing the initial condition of K to be zero, since the nonlinear static map P is set up in a way to map zero inputs to zero outputs. An important connection between the internal stability of F u (P, K) and the input-output stability of [P, K] has been stated in [2,Proposition 5]. Consequently, one may apply the small gain theorem for the convergence rate analysis of optimization methods.…”

Section: Figure 2: Feedback Interconnection With Exogenous Inputsmentioning

confidence: 99%

“…Our main approach is inspired by the loop transformation used in [2,5]. For any ρ ∈ (0, 1), the operators ρ + and ρ − are defined as the time-domain, time-dependent multipliers ρ k , ρ −k , respectively.…”

Section: Main Theoremmentioning

confidence: 99%

Control interpretations for first-order optimization methods

Lessard

2017

2017 American Control Conference (ACC)

Self Cite

View full text Add to dashboard Cite

First-order iterative optimization methods play a fundamental role in large scale optimization and machine learning. This paper presents control interpretations for such optimization methods. First, we give loop-shaping interpretations for several existing optimization methods and show that they are composed of basic control elements such as PID and lag compensators. Next, we apply the small gain theorem to draw a connection between the convergence rate analysis of optimization methods and the input-output gain computations of certain complementary sensitivity functions. These connections suggest that standard classical control synthesis tools may be brought to bear on the design of optimization algorithms.

show abstract

Exponential convergence bounds using integral quadratic constraints

Cited by 19 publications

References 30 publications

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

Robust and structure exploiting optimisation algorithms: an integral quadratic constraint approach

A Robust Accelerated Optimization Algorithm for Strongly Convex Functions

Control interpretations for first-order optimization methods

Contact Info

Product

Resources

About