Randomness and permutations in coordinate descent methods

Abstract

We consider coordinate descent (CD) methods with exact line search on convex quadratic problems. Our main focus is to study the performance of the CD method that use random permutations in each epoch and compare it to the performance of the CD methods that use deterministic orders and random sampling with replacement. We focus on a class of convex quadratic problems with a diagonally dominant Hessian matrix, for which we show that using random permutations instead of random with-replacement sampling improves the performance of the CD method in the worst-case. Furthermore, we prove that as the Hessian matrix becomes more diagonally dominant, the performance improvement attained by using random permutations increases. We also show that for this problem class, using any fixed deterministic order yields a superior performance than using random permutations. We present detailed theoretical analyses with respect to three different convergence criteria that are used in the literature and support our theoretical results with numerical experiments.

Fast Best Subset Selection: Coordinate Descent and Local Combinatorial Optimization Algorithms

“…Different CD methods have different operating characteristics under different settings -see e.g. [40,35,17] for discussions. In particular, since CCD and RCD do not require the evaluation of a full gradient in every step, their per-iteration cost is much smaller than the greedy CD.…”

Laplacian-based Semi-Supervised Learning in Multilayer Hypergraphs by Coordinate Descent

“…, m}. A random permutation of the variables every n iterations is also used, since it is known that this might lead to better practical performances in several cases (see, e.g., [28,65]).…”

On the global convergence of randomized coordinate gradient descent for non-convex optimization

“…We also refer to [26,27,33,47] for further convergence results for random coordinate selection for convex problems. More recently, convergence of methods with random permutation of coordinates (i.e., a random permutation of the d coordinates is used for every d step of the algorithm) have been analyzed, mostly for the case of quadratic objective functions [10,15,30,45]. It has been an ongoing research direction to compare various coordinate selection strategies in various settings.…”