Communication-efficient algorithms for statistical optimization

Abstract

We analyze two communication-efficient algorithms for distributed optimization in statistical settings involving large-scale data sets. The first algorithm is a standard averaging method that distributes the N data samples evenly to m machines, performs separate minimization on each subset, and then averages the estimates. We provide a sharp analysis of this average mixture algorithm, showing that under a reasonable set of conditions, the combined parameter achieves mean-squared error (MSE) that decays as O(N −1 + (N/m) −2 ). Whenever m ≤ √ N , this guarantee matches the best possible rate achievable by a centralized algorithm having access to all N samples. The second algorithm is a novel method, based on an appropriate form of bootstrap subsampling. Requiring only a single round of communication, it has mean-squared error that decays as O(N −1 + (N/m) −3 ), and so is more robust to the amount of parallelization. In addition, we show that a stochastic gradient-based method attains mean-squared error decaying as O(N −1 + (N/m) −3/2 ), easing computation at the expense of a potentially slower MSE rate. We also provide an experimental evaluation of our methods, investigating their performance both on simulated data and on a large-scale regression problem from the internet search domain. In particular, we show that our methods can be used to efficiently solve an advertisement prediction problem from the Chinese SoSo Search Engine, which involves logistic regression with N ≈ 2.4 × 10 8 samples and d ≈ 740,000 covariates.

Distributed optimization with arbitrary local solvers

“…At the other extreme, there are distributed methods using only a single round of communication, such as [24,36,38,80,81]. These methods require additional assumptions on the partitioning of the data, which are usually not satisfied in practice if the data are distributed "as is", i.e.…”

Distributed stochastic optimization and learning

“…When ERM is used and F (w) is λ-strongly convex, and f (w, z) is L-Lipschitz, H-smooth and has a J-Lipschitz Hessian, [29] obtain a guarantee onw of the following form (in expectation over the samples):…”

“…Optimizing over λ, the best that can be ensured from (13) for learning problems requiring regularization is therefor only a sample complexity that scales as 1/ 3 rather then 1/ 2 . If ERM is used on each machine [29] also suggested a bias-corrected approach that reduced the dependence on n in the second term to 1/n 3 , rather then 1/n 2 , but the problematic dependence on λ remains. These deficiencies are not only in the analysis.…”

Subsemble: an ensemble method for combining subset-specific algorithm fits

“…An average mixture (AVGM) procedure for fitting the parameter of a parametric model has been studied by [10]. AVGM partitions the full available dataset into disjoint subsets, estimates the parameter within each subset, and finally combines the estimates by simple averaging.…”