A control-theoretic perspective on optimal high-order optimization

Abstract

We provide a control-theoretic perspective on optimal tensor algorithms for minimizing a convex function in a finite-dimensional Euclidean space. Given a function $$\varPhi : {\mathbb {R}}^d \rightarrow {\mathbb {R}}$$ Φ : R d → R that is convex and twice continuously differentiable, we study a closed-loop control system that is governed by the operators $$\nabla \varPhi $$ ∇ Φ and $$\nabla ^2 \varPhi $$ ∇ 2 Φ together with a feedback control law $$\lambda (\cdot )$$ λ ( · ) satisfying the algebraic equation $$(\lambda (t))^p\Vert \nabla \varPhi (x(t))\Vert ^{p-1} = \theta $$ ( λ ( t ) ) p ‖ ∇ Φ ( x ( t ) ) ‖ p - 1 = θ for some $$\theta \in (0, 1)$$ θ ∈ ( 0 , 1 ) . Our first contribution is to prove the existence and uniqueness of a local solution to this system via the Banach fixed-point theorem. We present a simple yet nontrivial Lyapunov function that allows us to establish the existence and uniqueness of a global solution under certain regularity conditions and analyze the convergence properties of trajectories. The rate of convergence is $$O(1/t^{(3p+1)/2})$$ O ( 1 / t ( 3 p + 1 ) / 2 ) in terms of objective function gap and $$O(1/t^{3p})$$ O ( 1 / t 3 p ) in terms of squared gradient norm. Our second contribution is to provide two algorithmic frameworks obtained from discretization of our continuous-time system, one of which generalizes the large-step A-HPE framework of Monteiro and Svaiter (SIAM J Optim 23(2):1092–1125, 2013) and the other of which leads to a new optimal p-th order tensor algorithm. While our discrete-time analysis can be seen as a simplification and generalization of Monteiro and Svaiter (2013), it is largely motivated by the aforementioned continuous-time analysis, demonstrating the fundamental role that the feedback control plays in optimal acceleration and the clear advantage that the continuous-time perspective brings to algorithmic design. A highlight of our analysis is that we show that all of the p-th order optimal tensor algorithms that we discuss minimize the squared gradient norm at a rate of $$O(k^{-3p})$$ O ( k - 3 p ) , which complements the recent analysis in Gasnikov et al. (in: COLT, PMLR, pp 1374–1391, 2019), Jiang et al. (in: COLT, PMLR, pp 1799–1801, 2019) and Bubeck et al. (in: COLT, PMLR, pp 492–507, 2019).

Perseus: A Simple and Optimal High-Order Method for Variational Inequalities

“…Finally, we also note the existence of high-order methods obtained via discretization of continuoustime dynamical systems [Wibisono et al, 2016, Lin andJordan, 2021a]. In particular, Wibisono et al [2016] showed that the ACRN method and its p th -order variants can be derived from implicit discretization of an open-loop system without Hessian-driven damping.…”

“…In particular, Wibisono et al [2016] showed that the ACRN method and its p th -order variants can be derived from implicit discretization of an open-loop system without Hessian-driven damping. Lin and Jordan [2021a] have provided a control-theoretic perspective on p-order ANPE methods by recovering these methods from implicit discretization of a closed-loop system with Hessian-driven damping. Both of these works prove the convergence rate of p-order ACRN and ANPE methods via appeal to simple Lyapunov function arguments.…”

A Continuous-Time Perspective on Optimal Methods for Monotone Equation Problems

“…Most of the dynamical systems have been developed for modeling first-order algorithms for variational inequality and inclusion problems. The only exception we are aware of is Lin and Jordan [2021b] which extended the closed-loop damping approaches from convex optimization [Attouch et al, 2016a, Lin andJordan, 2021a] to monotone inclusion problems. However, the closed-loop control system in Lin and Jordan [2021b] fails to model the dual extrapolation method [Nesterov, 2007] and it remains unknown how the dual extrapolation step can exploit high-order smoothness.…”

“…From dynamical systems to high-order algorithms. Most of the high-order algorithms obtained via discretization of dynamical systems focus on convex optimization [Wibisono et al, 2016, Song et al, 2021, Lin and Jordan, 2021a, with an exception being recent work of Lin and Jordan [2021b] on monotone inclusion problems. In particular, Wibisono et al [2016] showed that the cubic regularization of Newton's methods and their variants [Nesterov and Polyak, 2006, Nesterov, 2008, 2021a] can be derived from the implicit discretization of the following open-loop system without Hessian-driven damping:…”

“…These approaches are, however, not suited for understanding and designing accelerated high-order algorithms in convex smooth optimization [Monteiro and Svaiter, 2013, Gasnikov et al, 2019, Alves, 2022. Recently, Lin and Jordan [2021a] presented an initial foray into the analysis of continuous-time dynamics of high-order algorithms using closed-loop control. Their approach can systematically generate discrete-time accelerated high-order algorithms, simplifying and generalizing the analysis in Monteiro and Svaiter [2013] using a unified Lyapunov function.…”