The Approximate Duality Gap Technique: A Unified Theory of First-Order Methods

Abstract

We present a general technique for the analysis of first-order methods. The technique relies on the construction of a duality gap for an appropriate approximation of the objective function, where the function approximation improves as the algorithm converges. We show that in continuous time enforcement of an invariant that this approximate duality gap decreases at a certain rate exactly recovers a wide range of first-order continuous-time methods. We characterize the discretization errors incurred by different discretization methods, and show how iteration-complexity-optimal methods for various classes of problems cancel out the discretization error. The techniques are illustrated on various classes of problems -including convex minimization for Lipschitz-continuous objectives, smooth convex minimization, composite minimization, smooth and strongly convex minimization, solving variational inequalities with monotone operators, and convex-concave saddle-point optimization -and naturally extend to other settings. 1 Width-independent algorithms enjoy poly-logarithmic dependence of their convergence times on the constraints matrix width -namely, the ratio between the constraint matrix maximum and minimum non-zero elements. By contrast, standard first-order methods incur (at best) linear dependence on the matrix width, which is not even considered to be polynomial-time convergence [27].

Understanding the acceleration phenomenon via high-resolution differential equations

“…In addition to the perspective of continuous-time dynamical systems, there has also been work on the acceleration from a control-theoretic point of view [11,24,25,31] and from a geometric point of view [15]. See also [18,19,21,23,32,37] for a number of other recent contributions to the study of the acceleration phenomenon.…”

From differential equation solvers to accelerated first-order methods for convex optimization

“…For the constrained problem (18), an accelerated forward-backward method is proposed in Algorithm 4. Both two algorithms call the proximal operation of g (over Q) only once in each iteration, and they are proved to share the same accelerated convergence rate (17).…”

Untitled

“…In this case, the algorithm can be improved: Yurii Nesterov showed how to do it using his accelerated gradient descent methods [19]. Recently there has been much interest in gaining a deeper understanding of this process, with proofs using "momentum" methods and continuous-time updates [29,24,17,30,8].…”

“…85-88] for a continuous perspective via Lyapunov functions. This is more explicit in recent papers [24,29,17,30,8] relating continuous and discrete updates to understand the acceleration phenomenon, e. g., Krichene et al [17] give the potential function we use in §5.2. However, these potential-function proofs and intuitions have not yet permeated into the commonly presented expositions.…”

“…One useful perspective is that of viewing these methods as discretizations of suitable continuous dynamics; this appears even in the classic work of Nemirovski and Yudin [18], and has been widely used recently (see, e. g., [24,29,17,30]). Another useful perspective is exhibit a "dual" lower bound on the optimal value via the convex conjugate, and use the duality gap to bound the error (see, e. g., [8,21]). We refer the interested reader to the respective papers for more details.…”