Parallel Coordinate Descent Methods for Big Data Optimization

Optimization Methods and Software

2017

Self Cite

We propose a novel stochastic gradient method-semi-stochastic coordinate descent (S2CD)-for the problem of minimizing a strongly convex function represented as the average of a large number of smooth convex functions: f (x) = 1 n i f i (x). Our method first performs a deterministic step (computation of the gradient of f at the starting point), followed by a large number of stochastic steps. The process is repeated a few times, with the last stochastic iterate becoming the new starting point where the deterministic step is taken. The novelty of our method is in how the stochastic steps are performed. In each such step, we pick a random function f i and a random coordinate j-both using nonuniform distributions-and update a single coordinate of the decision vector only, based on the computation of the j th partial derivative of f i at two different points. Each random step of the method constitutes an unbiased estimate of the gradient of f and moreover, the squared norm of the steps goes to zero in expectation, meaning that the stochastic estimate of the gradient progressively improves. The complexity of the method is the sum of two terms: O(n log(1/ǫ)) evaluations of gradients ∇f i and O(κ log(1/ǫ)) evaluations of partial derivatives ∇ j f i , whereκ is a novel condition number.

Section: Complexity Resultsmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Semi-stochastic coordinate descent

Konečný

Optimization Methods and Software

2017

Self Cite

“…

…”

mentioning

confidence: 99%

“…

In this work we study the parallel coordinate descent method (PCDM) proposed by Richtárik and Takáč [26] for minimizing a regularized convex function. We adopt elements from the work of Lu and Xiao [39], and combine them with several new insights, to obtain sharper iteration complexity results for PCDM than those presented in [26]. Moreover, we show that PCDM is monotonic in expectation, which was not confirmed in [26], and we also derive the first high probability iteration complexity result where the initial levelset is unbounded.

…”

mentioning

confidence: 99%

On the complexity of parallel coordinate descent

Tappenden

Takáč

Optimization Methods and Software

2017

Self Cite

Smooth Minimization of Nonsmooth Functions with Parallel Coordinate Descent Methods

Fercoq

Springer Proceedings in Mathematics &Amp; Statistics

2019

Self Cite

We study the performance of a family of randomized parallel coordinate descent methods for minimizing the sum of a nonsmooth and separable convex functions. The problem class includes as a special case L1-regularized L1 regression and the minimization of the exponential loss ("AdaBoost problem"). We assume the input data defining the loss function is contained in a sparse m × n matrix A with at most ω nonzeros in each row. Our methods need O(nβ/τ ) iterations to find an approximate solution with high probability, where τ is the number of processors and β = 1 + (ω − 1)(τ − 1)/(n − 1) for the fastest variant. The notation hides dependence on quantities such as the required accuracy and confidence levels and the distance of the starting iterate from an optimal point. Since β/τ is a decreasing function of τ , the method needs fewer iterations when more processors are used. Certain variants of our algorithms perform on average only O(nnz(A)/n) arithmetic operations during a single iteration per processor and, because β decreases when ω does, fewer iterations are needed for sparser problems.