Randomized Douglas-Rachford method for linear systems: Improved accuracy and efficiency

Abstract

The Douglas-Rachford (DR) method is a widely used method for finding a point in the intersection of two closed convex sets (feasibility problem). However, the method converges weakly and the associated rate of convergence is hard to analyze in general. In addition, the direct extension of the DR method for solving more-than-two-sets feasibility problems, called the r-sets-DR method, is not necessarily convergent. Recently, the introduction of randomization and momentum techniques in optimization algorithms has attracted increasing attention as it makes the algorithms more efficient. In this paper, we show that randomization is powerful for the DR method and can make the divergent r-sets-DR method converge. Specifically, we propose the randomized r-sets-DR (RrDR) method for solving the linear system Ax = b and prove its linear convergence in expectation, and the convergence rate does not depend on the dimension of the matrix A. We also study RrDR with heavy ball momentum and show that it converges at an accelerated linear rate. Numerical experiments are provided to confirm our results and demonstrate the significant improvement in accuracy and efficiency of the DR method that randomization and momentum bring.C i where C i := {x : a i , x = b i }.

On the relaxed greedy randomized Kaczmarz methods with momentum acceleration for solving matrix equation AXB=C

“…Polyak's momentum has been extended to solve the constrained and distributed optimization problems, confirming its performance advantages over standard gradient-based methods; see, e.g., [11,35]. In the context of solving liner system, Polyak's momentum technique has been spurred many related works by incorporating into various randomized iterative methods, e.g., randomized coordinate descent and Kaczmarz [22], sketch and project [22], sampling Kaczmarz Motzkin [23], randomized Douglas-Rachford [13], doubly stochastic iterative framework [14], and so on.…”

An Inertial Parametric Douglas–Rachford Splitting Method for Nonconvex Problems

“…Parameterization techniques to make DR algorithms more flexible in practice. Moreover, in the numerical experimental part of PDR, we observe that it saves a significant amount of running time compared to DR. Not only the PDR splitting method but much work (see [17][18][19]) on DR splitting methods has been done by adding parameters to its formulations, and it illustrates the superiority of parameterization.…”

“…Based on the inexact proximal point algorithm, Alves et al [22] combined inertial step and overrelaxation to propose a partially inexact inertial-relaxed Douglas-Rachford algorithm. Meanwhile, Han et al [19] study the randomized r-sets-Douglas-Rachford (RrDR) method with the inertial scheme and show that it converges at an accelerated linear rate. Additionally, Feng et al [23] developed an inertial Douglas-Rachford splitting (IDRS) method and demonstrated its validity in terms of signal recovery.…”

On the Polyak momentum variants of the greedy deterministic single and multiple row‐action methods

“…When 𝛽 = 0, this method resolves into the so-called gradient descent method. In the context of projection-based iterative methods, the Polyak momentum technique has been incorporated into various methods, for example, randomized coordinate descent and Kaczmarz (mRK), 25 randomized block Kaczmarz (mRBK) using the framework of sketch and project, 26 sampling Kaczmarz Motzkin, 27 randomized r-sets-Douglas-Rachford (mRrDR), 28 and doubly stochastic iterative framework. 26 For other related works, we refer to References 29 and 30 and the references therein.…”

“…IT (a) and CPU (b) versus m for mRK,25 mRrRD,28 mMWRK, and mFDBK when n = 150 in Example 1 with r = n∕10 and 𝜅 = n∕10.…”