Efficient Protocols for Distributed Classification and Optimization

Abstract

In distributed learning, the goal is to perform a learning task over data distributed across multiple nodes with minimal (expensive) communication. Prior work (Daumé III et al., 2012) proposes a general model that bounds the communication required for learning classifiers while allowing for ε training error on linearly separable data adversarially distributed across nodes.In this work, we develop key improvements and extensions to this basic model. Our first result is a two-party multiplicative-weight-update based protocol that uses O(d 2 log 1/ε) words of communication to classify distributed data in arbitrary dimension d, ε-optimally. This readily extends to classification over k nodes with O(kd 2 log 1/ε) words of communication. Our proposed protocol is simple to implement and is considerably more efficient than baselines compared, as demonstrated by our empirical results.In addition, we illustrate general algorithm design paradigms for doing efficient learning over distributed data. We show how to solve fixed-dimensional and high dimensional linear programming efficiently in a distributed setting where constraints may be distributed across nodes. Since many learning problems can be viewed as convex optimization problems where constraints are generated by individual points, this models many typical distributed learning scenarios. Our techniques make use of a novel connection from multipass streaming, as well as adapting the multiplicative-weight-update framework more generally to a distributed setting. As a consequence, our methods extend to the wide range of problems solvable using these techniques.

On Fundamental Limits of Robust Learning

“…In Section 3.5, we compare the above lower bound for half space learning with the bound given in [5], and we propose a communication-efficient distributed half-space learning algorithm, whose communication complexity is close to the above lower bound.…”

“…Several recent works on communication complexity for distributed PAC learning include [2,5,6] and there are also some works about distributed statistical estimation [25,7]. In particular, Duchi et al [8] demonstrated an analysis tool based on the information Fano inequalities.…”

“…The last decade has witnessed a tremendous growth in the amount of data involved in machine learning tasks. In many cases, data volume has outgrown the capacity of memory of a single machine and it is increasingly common that learning tasks are performed in a distributed fashion on many machines [2,5,25] or an online fashion on a single machine [15].…”

A Parallel algorithm for $\mathcal{X}$-Armed bandits

“…On the other hand, distributed stochastic optimization [11,12,13] and distributed PAC models [15,16,17] have also been intensively studied. Among them, Hillel et al [13] proposed the distributed algorithm for multi-armed bandits.…”

Distributed and Robust Support Vector Machine

“…In recent years, a significant amount of effort has been devoted to this problem ( [9,3,12,14,4,16,8,18,6]), and a number of distributed SVM algorithms with different strength have been developed. However most of them are still suffering from various limitations, such as high communication complexity, sub-optimal quality of solution, and slow convergence From a geometric point of view, training an SVM can be interpreted as finding a hyperplane that separates two classes of points while maximizing the separating margin.…”