Distributed optimization with arbitrary local solvers

Abstract

With the growth of data and necessity for distributed optimization methods, solvers that work well on a single machine must be re-designed to leverage distributed computation. Recent work in this area has been limited by focusing heavily on developing highly specific methods for the distributed environment. These special-purpose methods are often unable to fully leverage the competitive performance of their well-tuned and customized single machine counterparts. Further, they are unable to easily integrate improvements that continue to be made to single machine methods. To this end, we present a framework for distributed optimization that both allows the flexibility of arbitrary solvers to be used on each (single) machine locally and yet maintains competitive performance against other state-of-the-art special-purpose distributed methods. We give strong primal-dual convergence rate guarantees for our framework that hold for arbitrary local solvers. We demonstrate the impact of local solver selection both theoretically and in an extensive experimental comparison. Finally, we provide thorough implementation details for our framework, highlighting areas for practical performance gains.Keywords: primal-dual algorithm; distributed computing; machine learning; convergence analysis 2010 Mathematics Subject Classification: 68W15; 68W20; 68W10; 68W40 MotivationRegression and classification techniques, represented in the general class of regularized loss minimization problems [71], are among the most central tools in modern big data analysis, machine learning, and signal processing. For these tasks, much effort from both industry and academia has gone into the development of highly tuned and customized solvers. However, with the massive growth of available datasets, major roadblocks still persist in the distributed setting, where data no longer fit in the memory of a single computer, and computation must be split across multiple machines in a network [3,7,12,18,22,29,32,34,37,46,52,62,64,67,78].On typical real-world systems, communicating data between machines is several orders of magnitude slower than reading data from main memory, e.g. when leveraging commodity hardware. Therefore when trying to translate existing highly tuned single machine solvers to the *Corresponding author. Email: takac.mt@gmail.com 814C. Ma et al. distributed setting, great care must be taken to avoid this significant communication bottleneck [26,74].While several distributed solvers for the problems of interest have been recently developed, they are often unable to fully leverage the competitive performance of their tuned and customized single machine counterparts, which have already received much more research attention. More importantly, it is unfortunate that distributed solvers cannot automatically benefit from improvements made to the single machine solvers, and therefore are forced to lag behind the most recent developments.In this paper, we make a step towards resolving these issues by proposing a general communication-efficient distribu...

Adaptive Federated Learning in Resource Constrained Edge Computing Systems

“…Combining these two cases, we have proven (36). In the following, we define c c ν and b b ν for simplicity.…”

Scheduling Policies for Federated Learning in Wireless Networks

“…Based on the solution, the UE updates the local reference a t [k] per (18), and if being selected by the AP, it sends out a global update ∆v t k via the allocated subchannel. • At the AP side, it selects a subgroup of UEs for update collection, decodes the received packet, and performs a global aggregation according to (20). The new global parameter is redistributed to all the associated UEs using an error free channel.…”

Distributed block-diagonal approximation methods for regularized empirical risk minimization

“…Therefore, ξ i (X T i w(α)) also has a bounded domain, indicating that by compactness we can find L ≥ 0 such that this continuous function is Lipschitz continuous within this domain. Moreover, the formulation (32) satisfies the form (29), so (21) holds by Lemma 1. Therefore, Assumption 3 is satisfied.…”

“…The difference is more significant in the SVM problems, showing that exact line search has its advantage over backtracking, especially because its cost is low, while backtracking is still better than the fixed step size scheme. The reason behind is that although the approach of DisDCA provides a safe upper bound modeling of the objective difference such that the local updates can be directly applied to ensure the objective decrease, this upper bound might be too conservative as suggested in [29], but more aggressive upper bound modelings might be computationally impractical to obtain. On the other hand, our approach provides an efficient way to dynamically estimate how aggressive the updates can be, depending to the current iterate.…”