Random reordering in SOR-type methods

Oswald, Peter; Zhou, Weiqi

doi:10.1007/s00211-016-0829-7

Cited by 17 publications

(30 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, Oswald and Zhou [19] analyzed the effects of random permutations for the successive over-relaxation (SOR) method, which is equivalent to the CD method with exact line search for a particular choice of algorithm parameter. They consider quadratic problems whose Hessian matrix is positive semidefinite and present convergence guarantees for SOR iterations with random permutations, which implies the following guarantee on the performance of RPCD.…”

Section: Prior Work On CD Methods With Random Permutationsmentioning

confidence: 99%

Randomness and permutations in coordinate descent methods

et al. 2019

View full text Add to dashboard Cite

We consider coordinate descent (CD) methods with exact line search on convex quadratic problems. Our main focus is to study the performance of the CD method that use random permutations in each epoch and compare it to the performance of the CD methods that use deterministic orders and random sampling with replacement. We focus on a class of convex quadratic problems with a diagonally dominant Hessian matrix, for which we show that using random permutations instead of random with-replacement sampling improves the performance of the CD method in the worst-case. Furthermore, we prove that as the Hessian matrix becomes more diagonally dominant, the performance improvement attained by using random permutations increases. We also show that for this problem class, using any fixed deterministic order yields a superior performance than using random permutations. We present detailed theoretical analyses with respect to three different convergence criteria that are used in the literature and support our theoretical results with numerical experiments.

show abstract

Section: Prior Work On CD Methods With Random Permutationsmentioning

confidence: 99%

Randomness and permutations in coordinate descent methods

et al. 2019

View full text Add to dashboard Cite

show abstract

“…The Itoh-Abe discrete gradient scheme (19) is equivalent to the SOR method. This subsection explains the equivalence.…”

Section: Equivalence Between the Scheme Given By (19) And The Sor Methodsmentioning

confidence: 99%

“…, a nn ), and L and U are strictly lower and upper triangular n × n matrices, respectively. The Itoh-Abe discrete gradient scheme (19) is a stationary iterative method of the form…”

Section: Equivalence Between the Scheme Given By (19) And The Sor Methodsmentioning

confidence: 99%

On the equivalence between SOR-type methods for linear systems and the discrete gradient methods for gradient systems

Miyatake

Sogabe

Zhang

2018

Journal of Computational and Applied Mathematics

View full text Add to dashboard Cite

The iterative nature of many discretisation methods for continuous dynamical systems has led to the study of the connections between iterative numerical methods in numerical linear algebra and continuous dynamical systems. Certain researchers have used the explicit Euler method to understand this connection, but this method has its limitation. In this study, we present a new connection between successive over-relaxation (SOR)-type methods and gradient systems; this connection is based on discrete gradient methods. The focus of the discussion is the equivalence between SOR-type methods and discrete gradient methods applied to gradient systems. The discussion leads to new interpretations for SOR-type methods. For example, we found a new way to derive these methods; these methods monotonically decrease a certain quadratic function and obtain a new interpretation of the relaxation parameter. We also obtained a new discrete gradient while studying the new connection.

show abstract

“…x r+1 π(1) = (1 − α)x r π(1) + ατ x r π(2) , x r+1 π(2) = (1 − α)x r π(2) + ατ x r+1 π (1) . (7) Note that this method is referred to as the shuffled SOR in [21].…”

Section: Background On the Convergence Of G-s Type Algorithmmentioning

confidence: 99%

“…Hence, the convergence of the SOR algorithm is guaranteed if the spectral norm of the iteration matrix is strictly less than one. Recently, the work [21] shows that when A is symmetric and positive semidefinite (PSD), then the G-S algorithm with RP rule can yield better convergence rate compared with the cyclic G-S (in the asymptotic region where n is large). From the above discussion it is clear that the classical G-S type algorithm does not work for any matrix A.…”

Section: Background On the Convergence Of G-s Type Algorithmmentioning

confidence: 99%

A linearly convergent doubly stochastic Gauss–Seidel algorithm for solving linear equations and a certain class of over-parameterized optimization problems

et al. 2019

View full text Add to dashboard Cite

Consider the classical problem of solving a general linear system of equations Ax = b. It is well known that the (successively over relaxed) Gauss-Seidel scheme and many of its variants may not converge when A is neither diagonally dominant nor symmetric positive definite. Can we have a linearly convergent G-S type algorithm that works for any A? In this paper we answer this question affirmatively by proposing a doubly stochastic G-S algorithm that is provably linearly convergent (in the mean square error sense) for any feasible linear system of equations. The key in the algorithm design is to introduce a nonuniform double stochastic scheme for picking the equation and the variable in each update step as well as a stepsize rule. These techniques also generalize to certain iterative alternating projection algorithms for solving the linear feasibility problem Ax ≤ b with an arbitrary A, as well as high-dimensional minimization problems for training over-parameterized models in machine learning. Our results demonstrate that a carefully designed randomization scheme can make an otherwise divergent G-S algorithm converge. * equal contributions. M. Razaviyayn is with the

show abstract

Random reordering in SOR-type methods

Cited by 17 publications

References 15 publications

Randomness and permutations in coordinate descent methods

Randomness and permutations in coordinate descent methods

On the equivalence between SOR-type methods for linear systems and the discrete gradient methods for gradient systems

A linearly convergent doubly stochastic Gauss–Seidel algorithm for solving linear equations and a certain class of over-parameterized optimization problems

Contact Info

Product

Resources

About