Local linear convergence analysis of Primal–Dual splitting methods

Abstract

In this paper, we study the local linear convergence properties of a versatile class of Primal-Dual splitting methods for minimizing composite non-smooth convex optimization problems. Under the assumption that the non-smooth components of the problem are partly smooth relative to smooth manifolds, we present a unified local convergence analysis framework for these Primal-Dual splitting methods. More precisely, in our framework we first show that (i) the sequences generated by Primal-Dual splitting methods identify a pair of primal and dual smooth manifolds in a finite number of iteration, and then (ii) enter a local linear convergence regime, which is for instance characterized in terms of the structure of the underlying active smooth manifolds. We also show how our results for Primal-Dual splitting specialize to cover existing one on Forward-Backward splitting and Douglas-Rachford splitting/ADMM (alternating direction methods of multipliers). Moreover, based on these obtained local convergence analysis result, several practical acceleration techniques for the class of Primal-Dual splitting methods are discussed. To exemplify the usefulness of the obtained result, we consider several concrete numerical experiments arising from applicative fields including signal/image processing, inverse problems and machine learning, etc. The demonstration not only verify the local linear convergence behaviour of Primal-Dual splitting methods, but also the insights on how to accelerate them in practice. Primal-Dual splitting methodsPrimal-Dual splitting methods to solve more or less complex variants of (P P )-(P D ) have witnessed a recent wave of interest in the literature [13,11,50,18,29,21,16]. All these approaches achieve full splitting, they involve the resolvents of R and J * , the gradients of F and G * and the linear operator L, all separately at various points in the course of iteration. For instance, building on the seminal work of [3], the now-popular scheme proposed in [13] solves (P P )-(P D ) with F = G * = 0. The authors in [29] have shown that the Primal-Dual splitting method of [13] can be seen as a proximal point algorithm (PPA) in R n × R m endowed with a suitable norm. Exploiting the same idea, the author in [21] considered (P P ) with G * = 0, and proposed an iterative scheme which can be interpreted as a Forward-Backward (FB) splitting again with an appropriately renormed space. This idea is further extended in [50] to solve more complex problems such as that in (P P ). A variable metric version was proposed in [19]. Motivated by the structure of (1.1), [11] and [18] proposed a Forward-Backward-Forward scheme [48] to solve it.In this paper, we will focus the unrelaxed Primal-Dual splitting method summarized in Algorithm 1. This scheme covers that of [13,50,29,21,16]. Though we omit the details here for brevity, our analysis and conclusions carry through to the method proposed in [11,18]. Algorithm 1: A Primal-Dual splitting method(1.2) k = k + 1; until convergence;See Section A.2 for the proof. Remark 3.4....

Nearly Optimal Linear Convergence of Stochastic Primal-Dual Methods for Linear Programming

“…To deal with this issue, [20] presents a variant of ADMM and shows the linear convergence of the proposed method for LP. More recently [33,34] show that many primal-dual algorithms under a mild non-degeneracy condition have eventual linear convergence, but it may take a long time before reaching the linear convergence regime. [6] propose a restarted scheme for LP in the primal-dual formulation.…”

Distributed Computation of Stochastic GNE with Partial Information: An Augmented Best-Response Approach

“…As we have previously mentioned, the fixed point iteration discussed in this paper engages two reflected resolvent operators, which merely admit quasinonexpansiveness rather than contractiveness. The convergence rate analysis under this setting remains an under-explored yet increasingly active direction [44], [45].…”

Convergence Rates of Forward–Douglas–Rachford Splitting Method

“…Recently, local linear convergence of operator splitting algorithms for optimization has recently attracted a lot of attention; see [15] for Forward–Backward-type methods, [16] for Douglas–Rachford splitting, and [17] for Primal–Dual splitting algorithms. This work particularly exploits the underlying geometric structure of the optimization problems, achieving a local linear convergence result without assuming conditions like strong convexity, unlike what is proved in [8, 18].…”

“…The key idea here is to exploit the geometry of the underlying objective around its minimizers. This has been done for instance in [15–17, 19] for the FB scheme, Douglas–Rachford splitting/ADMM and Primal–Dual splitting, under the umbrella of partial smoothness. The error bound property,1 as highlighted in the seminal work of [22, 23], is used by several authors to study linear convergence of first-order descent-type algorithms, and in particular FB splitting; see, for example, [20, 21, 24, 25].…”