Principled Analyses and Design of First-Order Methods with Inexact Proximal Operators

Abstract

Proximal operations are among the most common primitives appearing in both practical and theoretical (or high-level) optimization methods. This basic operation typically consists in solving an intermediary (hopefully simpler) optimization problem. In this work, we survey notions of inaccuracies that can be used when solving those intermediary optimization problems. Then, we show that worst-case guarantees for algorithms relying on such inexact proximal operations can be systematically obtained through a generic procedure based on semidefinite programming. This methodology is primarily based on the approach introduced by Drori and Teboulle [36] and on convex interpolation results, and allows producing non-improvable worst-case analyzes. In other words, for a given algorithm, the methodology generates both worst-case certificates (i.e., proofs) and problem instances on which those bounds are achieved.Relying on this methodology, we provide three new methods with conceptually simple proofs: (i) an optimized relatively inexact proximal point method, (ii) an extension of the hybrid proximal extragradient method of Monteiro and Svaiter [73], and (iii) an inexact accelerated forward-backward splitting supporting backtracking line-search, and both (ii) and (iii) supporting possibly strongly convex objectives. Finally, we use the methodology for studying a recent inexact variant of the Douglas-Rachford splitting due to Eckstein and Yao [44].We showcase and compare the different variants of the accelerated inexact forward-backward method on a factorization and a total variation problem.

Optimal First-Order Algorithms as a Function of Inequalities

“…The performance estimation problem (PEP) is a computer-assisted proof methodology that analyzes the worst-case performance of optimization algorithms through semidefinite programs (Drori and Teboulle, 2014;Taylor et al, 2017b,a). The use of the PEP has lead to many discoveries that would have otherwise been difficult without the assistance (Kim and Fessler, 2018a;Taylor et al, 2018;Taylor and Bach, 2019;Barré et al, 2020;De Klerk et al, 2020;Gu and Yang, 2020;Lieder, 2021;Ryu et al, 2020;Dragomir et al, 2021;Kim, 2021;Yoon and Ryu, 2021). Notably, the algorithms OGM (Drori and Teboulle, 2014;Kim andFessler, 2016, 2018b), OGM-G (Kim andFessler, 2021), andITEM (Taylor and were obtained by using the PEP for the setup of minimizing a smooth convex (possibly strongly convex) function.…”

Variants of the A-HPE and large-step A-HPE algorithms for strongly convex problems with applications to accelerated high-order tensor methods

“…When combined with restart techniques, improved rates for the uniformly-and/or strongly-convex case were also obtained in [3,9] (see also [12]). The A-HPE for strongly-convex problems was also recently studied in [5] within the framework of "performance estimation problems (PEPs)" (see remark (iv) following Algorithm 1). We also mention that local superlinear convergence rates for tensor methods were obtained in [7].…”

“…(iv) We also mention that Algorithm 1 is closely related to a variant of the A-HPE for strongly convex objectives presented and studied in [5,Section 4.2]. However, in constrast to the analysis in [5], which is supported on "performance estimation problems (PEPs)", in this contribution we take an approach similar to the one which was taken in [18,23]. In doing so, we obtain global convergence rates for Algorithm 1 in terms of function values, sequences and (sub-)gradients (see Theorems 2.6 and 2.9).…”

“…Among them, inexact proximal-point methods based on relative-error criterion for the subproblems are currently quite popular. For the more abstract setting of solving inclusions for maximal monotone operators, this approach was initially developed by Solodov and Svaiter (see, e.g., [26,27,28,29]), subsequently studied, from the viewpoint of computational complexity, by Monteiro and Svaiter (see, e.g., [15,16,17,18]) and has gained a lot of attention by different authors and research groups (see, e.g., [4,5,8,11,13]) with many applications in optimization algorithms and related topics such as variational inequalities, saddle-point problems, etc. The starting point of this contribution is [18], where the relative-error inexact hybrid proximal extragradient (HPE) method [16,26] was accelerated for convex optimization, by using Nesterov's acceleration [19].…”

“…The resulting accelerated HPE-type algorithms, called A-HPE and large-step A-HPE, were applied to first-and second-order optimization, with iteration-complexities O 1/k 2 and O 1/k 7/2 , respectively. The A-HPE and/or the large-step A-HPE algorithms were recently studied also in [3,5,6,9,11,13], with applications in high-order optimization, machine learning and tensor methods.…”