Accelerated, Parallel, and Proximal Coordinate Descent

Abstract

Abstract. We propose a new randomized coordinate descent method for minimizing the sum of convex functions each of which depends on a small number of coordinates only. Our method (APPROX) is simultaneously Accelerated, Parallel, and PROXimal; this is the first time such a method is proposed. In the special case when the number of processors is equal to the number of coordinates, the method converges at the rate 2ωLR 2 /(k + 1) 2 , where k is the iteration counter,ω is a data-weighted average degree of separability of the loss function,L is the average of Lipschitz constants associated with the coordinates and individual functions in the sum, and R is the distance of the initial point from the minimizer. We show that the method can be implemented without the need to perform full-dimensional vector operations, which is the major bottleneck of accelerated coordinate descent. The fact that the method depends on the average degree of separability, and not on the maximum degree, can be attributed to the use of new safe large stepsizes, leading to improved expected separable overapproximation (ESO). These are of independent interest and can be utilized in all existing parallel randomized coordinate descent algorithms based on the concept of ESO. In special cases, our method recovers several classical and recent algorithms such as simple and accelerated proximal gradient descent, as well serial, parallel, and distributed versions of randomized block coordinate descent. Our bounds match or improve on the best known bounds for these methods.

On optimal probabilities in stochastic coordinate descent methods

“…Parallel methods were considered in [2,19,21], and more recently in [1,5,6,12,13,25,27,28]. A memory distributed method scaling to big data problems was recently developed in [22].…”

“…The first assumption generalizes the ESO concept introduced in [21] and later used in [5,6,22,27,28] to nonuniform samplings. The second assumption requires that φ be strongly convex.…”

“…In this sense, the assumption is not restrictive. Inequalities of the type (2), in the uniform case ( p i = p j for all i, j), were studied in [6,21,22,27]. Motivated by the introduction of the nonuniform ESO assumption in this paper, and the development in Sect.…”

A delayed proximal gradient method with linear convergence rate

“…Nesterov's results have since been extended in several directions, for both randomized and deterministic update orders, (see, e.g. [2][3][4][5]). …”

Asynchronous incremental block-coordinate descent

“…Their code, called HOGWILD!, uses atomic operations to A. Aytekin, H. R. Feyzmahdavian avoid locking of loosely coupled memory locations in the minimization problem of sparse separable cost functions, and have achieved linear speedup in the number of processors. Following a similar sparsity and separability assumption, Fercoq and Richtárik have proposed an accelerated, parallel and proximal coordinate descent method to better utilize the available processors to achieve even further speedups [6].…”