Geometry of First-Order Methods and Adaptive Acceleration

Abstract

First-order operator splitting methods are ubiquitous among many fields through science and engineering, such as inverse problems, signal/image processing, statistics, data science and machine learning, to name a few. In this paper, we study a geometry property of first-order methods when applying to solve non-smooth optimization problems. With the tool "partial smoothness", we design a framework to analyze the trajectory of the fixed-point sequence generated by first-order methods and show that locally the fixed-point sequence settles onto a regular trajectory such as straight line or spiral. Based on this finding, we discuss the limitation of current widely used "inertial acceleration" technique, and propose a trajectory following adaptive acceleration algorithm. Global convergence is established for the proposed acceleration scheme based on the perturbation of fixed-point iteration. Locally, we first build connections between the acceleration scheme and the well studied "vector extrapolation technique" in the field of numerical analysis, and then discuss acceleration guarantees of the proposed acceleration scheme. Moreover, our result provides a geometric interpretation of vector extrapolation. Numeric experiments on various first-order methods are provided to demonstrate the advantage of the proposed adaptive acceleration scheme.

Anderson acceleration of coordinate descent

“…However, similarly to Mai and Johansson (2019) and Poon and Liang (2020) we observed that regularizing the linear system does not seem necessary, and can even hurt the convergence speed. Figure 5 shows the influence of the regularization parameter on the convergence on the rcv1 dataset for a sparse logistic regression problem, with K = 5 and λ = λ max /30.…”

“…We compare multiple algorithms to solve popular Machine Learning problems: the Lasso, the elastic net, and sparse logistic regression (experiments on group lasso are in Appendix A.5). The compared algorithms are the following: proximal gradient descent (PGD, Combettes and Wajs 2005), Nesterov-like inertial PGD (FISTA, Beck and Teboulle 2009), Anderson accelerated PGD (Mai and Johansson, 2019;Poon and Liang, 2020), proximal coordinate descent (PCD, Tseng and Yun 2009), inertial PCD (Lin et al, 2014;Fercoq and Richtárik, 2015), Anderson accelerated PCD (ours). We use datasets from libsvm (Fan et al, 2008) and openml (Feurer et al, 2019) (Table 1), varying as much as possible to demonstrate the versatility of our approach.…”

“…Interestingly, numerical performances still show significant improvements on nonquadratic objectives. Anderson acceleration has been adapted to various algorithms such as Douglas-Rachford (Fu et al, 2019), ADMM (Poon and Liang, 2019) or proximal gradient descent (Zhang et al, 2018;Mai and Johansson, 2019;Poon and Liang, 2020). Among main benefits, the practical version of // does not affect x (k) 6 return x…”

Two-step Inertial Method for Solving Split Common Null Point Problem With Multiple Output Sets in Hilbert Spaces

“…It was shown in [30,Section 4] by example that one-step inertial extrapolation w n = x n +θ(x n −x n−1 ), θ ∈ [0, 1) may fail to provide acceleration. It was remarked in [24,Chapter 4] that the use of inertial of more than two points x n , x n−1 could provide acceleration.…”

Convergence results of two-step inertial proximal point algorithm

“…Advantages of two-step proximal point algorithms. In [47,48], Poon and Liang discussed some limitations of inertial Douglas-Rachford splitting method and inertial ADMM. For example, consider the following feasibility problem in…”