Analysis of Optimization Algorithms via Integral Quadratic Constraints: Nonstrongly Convex Problems

Fazlyab, Mahyar; Ribeiro, Alejandro; Morari, Manfred; Preciado, Víctor M.

doi:10.1137/17m1136845

Cited by 99 publications

(140 citation statements)

References 31 publications

Supporting

Mentioning

134

Contrasting

Unclassified

Order By: Relevance

“…To overcome this challenge, we present a modified semidefinite program that uses more general Lyapunov functions which are obtained by augmenting standard quadratic terms with the objective function. This type of generalized Lyapunov functions has been introduced in [46], [49] and used to study convergence of optimization algorithms for non-strongly convex problems. We employ a modified SDP to derive meaningful upper bounds on J in (3) for Nesterov's method as well.…”

Section: General Strongly Convex Problemsmentioning

confidence: 99%

See 1 more Smart Citation

Variance Amplification of Accelerated First-Order Algorithms for Strongly Convex Quadratic Optimization Problems

Mohammadi

Razaviyayn

Jovanović

2018

2018 IEEE Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

We study the robustness of accelerated first-order algorithms to stochastic uncertainties in gradient evaluation.Specifically, for unconstrained, smooth, strongly convex optimization problems, we examine the mean-square error in the optimization variable when the iterates are perturbed by additive white noise. This type of uncertainty may arise in situations where an approximation of the gradient is sought through measurements of a real system or in a distributed computation over network. Even though the underlying dynamics of first-order algorithms for this class of problems are nonlinear, we establish upper bounds on the mean-square deviation from the optimal value that are tight up to constant factors. Our analysis quantifies fundamental trade-offs between noise amplification and convergence rates obtained via any acceleration scheme similar to Nesterov's or heavy-ball methods. To gain additional analytical insight, for strongly convex quadratic problems we explicitly evaluate the steady-state variance of the optimization variable in terms of the eigenvalues of the Hessian of the objective function. We demonstrate that the entire spectrum of the Hessian, rather than just the extreme eigenvalues, influence robustness of noisy algorithms. We specialize this result to the problem of distributed averaging over undirected networks and examine the role of network size and topology on the robustness of noisy accelerated algorithms. the steady-state mean error in the objective value and it was shown that, for the same convergence rate, Nesterov-like method with properly-selected parameters can be more robust than gradient descent. This is not surprising because gradient descent can be viewed as a special case of Nesterov's method with a zero momentum parameter. This observation was used in [41] to design an optimal multi-stage algorithm that does not require information about variance of the noise. In this paper, however, we focus on the variance amplification of the iterates (and not the function value) and discuss the connections between the two robustness measures. We show that any choice of parameters for Nesterov's or heavy-ball methods that yields an accelerated convergence rate increases variance amplification relative to gradient descent. More precisely, for the problem with the condition number κ, an algorithm with accelerated convergence rate of at least 1 − c/ √ κ, where c is a positive constant, increases the variance amplification in the iterates by a factor of √ κ. The robustness problem was also studied in [42] where the authors show similar behavior of Nesterov's method and gradient descent in an asymptotic regime in which the stepsize May 28, 2019 DRAFT 4) We extend our analysis from quadratic objective functions to general strongly convex problems. We borrow an approach based on linear matrix inequalities from control theory to establish upper bounds on the noise amplification of both gradient descent and Nesterov's accelerated algorithm. Furthermore, for any given condition number, we demonstrate th...

show abstract

Section: General Strongly Convex Problemsmentioning

confidence: 99%

“…to further tighten the constraints on the gradient ∇f and reduce conservativeness. In what follows, we build on the results of [46] and present an alternative LMI in Lemma 2 that is obtained using a Lyapunov function of the form…”

Section: General Strongly Convex Problemsmentioning

confidence: 99%

Variance Amplification of Accelerated First-Order Algorithms for Strongly Convex Quadratic Optimization Problems

Mohammadi

Razaviyayn

Jovanović

2018

2018 IEEE Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

show abstract

“…Note that the matrix inequality in (15) is linear in all the parameters except for α and λ. We can use the same technique as shown in Section III-B to transform (15) into an LMI when the stepsize α is fixed. Let…”

Section: Linear Convergence Of Tosmentioning

confidence: 99%

“…Then the same methods in the proof Appendix A-A and A-B apply here. If (15) holds, then v T k W 2 v k ≤ 0, which means V k+1 ≤ ρ 2 V k and linear convergence (16) holds.…”

Section: Proof Of Theoremmentioning

confidence: 99%

See 1 more Smart Citation

Robust Convergence Analysis of Three-Operator Splitting

Wang

Fazlyab

Chen

et al. 2019

2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

Self Cite

View full text Add to dashboard Cite

Operator splitting methods solve composite optimization problems by breaking them into smaller sub-problems that can be solved sequentially or in parallel. In this paper, we propose a unified framework for certifying both linear and sublinear convergence rates for three-operator splitting (TOS) method under a variety of assumptions about the objective function. By viewing the algorithm as a dynamical system with feedback uncertainty (the oracle model), we leverage robust control theory to analyze the worst-case performance of the algorithm using matrix inequalities. We then show how these matrix inequalities can be used to verify sublinear/linear convergence of the TOS algorithm and guide the search for selecting the parameters of the algorithm (both symbolically and numerically) for optimal worst-case performance. We illustrate our results numerically by solving an input-constrained optimal control problem.

show abstract

A fixed step distributed proximal gradient push‐pull algorithm based on integral quadratic constraint

Gao

Xie

Ren

2023

Optim Control Appl Methods

View full text Add to dashboard Cite

In order to solve the distributed optimization problem with smooth + nonsmooth structure of the objective function on unbalanced directed networks, this article uses the proximal operator to deal with the nonsmooth part of the objective function, and designs and analyzes the fixed step proximal gradient Push‐Pull (PG‐Push‐Pull) algorithm. Firstly, the Integral Quadratic Constraint (IQC) suitable for proximal gradient Push‐Pull algorithm is given. When the smooth part of the objective function is strongly convex and the gradient satisfies the Lipchitz condition, the convergence of the algorithm is proved, and the convergence analysis is transformed into solving a linear matrix inequality by using this IQC framework. Its feasibility can ensure that the proposed algorithm has linear convergence rate, which is the same as that of Push‐Pull gradient algorithm. Then, the upper bound of convergence rate can be found by solving a Non‐Linear Programming problem. Finally, an example is given to analyze the upper bound of the convergence rate and verify the effectiveness of the proposed algorithm.

show abstract

Analysis of Optimization Algorithms via Integral Quadratic Constraints: Nonstrongly Convex Problems

Cited by 99 publications

References 31 publications

Variance Amplification of Accelerated First-Order Algorithms for Strongly Convex Quadratic Optimization Problems

Variance Amplification of Accelerated First-Order Algorithms for Strongly Convex Quadratic Optimization Problems

Robust Convergence Analysis of Three-Operator Splitting

A fixed step distributed proximal gradient push‐pull algorithm based on integral quadratic constraint

Contact Info

Product

Resources

About