Straggler Resilient Serverless Computing Based on Polar Codes

Bartan, Burak; Pilancı, Mert

doi:10.1109/allerton.2019.8919767

Cited by 11 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lemma 3 (Bounds on Z m for large kernels): Let K be a kernel selected uniformly at random from all ℓ×ℓ non-singular binary matrices, and let Z m be the random process defined in (6) with initial condition Z 0 = ǫ. Fix a small constant δ > 0.…”

Section: Extension To Polar Codes With Large Kernelsmentioning

confidence: 99%

“…In [2], maximum distance separable (MDS) codes are used to speed up matrix multiplication, and for matrix multiplication in a large finite field polynomial codes are employed in [3]. Luby Transform (LT) codes are proposed in [4], [5] for coded computation, and the application of polar codes to serverless computing is considered in [6], [7]. In [8], the average execution time of a coded computing system is related to the error probability of the linear code used to add redundancy.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Polar Coded Computing: The Role of the Scaling Exponent

Fathollahi¹,

Mondelli²

2022

Preprint

View full text Add to dashboard Cite

We consider the problem of coded distributed computing using polar codes. The average execution time of a coded computing system is related to the error probability for transmission over the binary erasure channel in recent work by Soleymani, Jamali and Mahdavifar, where the performance of binary linear codes is investigated. In this paper, we focus on polar codes and unveil a connection between the average execution time and the scaling exponent µ of the family of codes. The scaling exponent has emerged as a central object in the finitelength characterization of polar codes, and it captures the speed of convergence to capacity. In particular, we show that (i) the gap between the normalized average execution time of polar codes and that of optimal MDS codes is O(n −1/µ ), and (ii) this upper bound can be improved to roughly O(n −1/2 ) by considering polar codes with large kernels. We conjecture that these bounds could be improved to O(n −2/µ ) and O(n −1 ), respectively, and provide a heuristic argument as well as numerical evidence supporting this view.

show abstract

Section: Extension To Polar Codes With Large Kernelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Polar Coded Computing: The Role of the Scaling Exponent

Fathollahi¹,

Mondelli²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The application of traditional Polar Codes to distributed computation was first proposed in our recent work [9].…”

Section: B Prior Workmentioning

confidence: 99%

“…The decoding of the overall computation can be done in a similar spirit to the successive cancellation decoder of traditional Polar Codes over finite fields, where a major difference is that it operates over real-valued data. An implementation of the successive decoding strategy for real-valued polar codes was described in the earlier work [9]. Specifically, the principle behind the decoder parallels the successive cancellation strategy described in [8] and is as follows.…”

Section: Decodingmentioning

confidence: 99%

Computational Polarization: An Information-theoretic Method for Resilient Computing

Pilancı¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We introduce an error resilient distributed computing method based on an extension of the channel polarization phenomenon to distributed algorithms. The method leverages an algorithmic split operation that transforms two identical compute nodes to slow and fast workers, which parallels the channel split operation in Polar Codes. This operation preserves the average runtime, analogous to the conservation of Shannon capacity in channel polarization.By leveraging a recursive construction in a similar spirit to the Fast Fourier Transform, this method synthesizes virtual compute nodes with dispersed return time distributions, which we call computational polarization. We show that the runtime distributions form a functional martingale processes, identify their limiting distributions in closedform expressions together with non-asymptotic convergence rates, and prove strong convergence results in Banach spaces. We provide an information-theoretic lower bound on the overall runtime of any coded computation method and show that the computational polarization approach asymptotically achieves the optimal runtime for computing linear functions. An important advantage is the near linear time decoding procedure, which is significantly cheaper than Maximum Distance Separable codes. IndexTerms error correcting codes, Polar codes, coding for computation, random processes, martingales in Banach spaces I. INTRODUCTION As a result of the recent growth of data, the computing paradigm has shifted into massively distributed computing systems. Several distributed architectures and software frameworks have been developed for large scale computational problems. Notable examples include the open source distributed computing framework Apache Spark [1] and

show abstract

“…Naturally, the communication between the high-latency storage and the commodity workers is extremely slow [e.g., see 19], resulting in impractical end-to-end times for many popular optimization algorithms such as SGD [15,16]. Furthermore, communication failures between the cloud storage and serverless workers consistently give rise to stragglers, and this introduces synchronization delays [20,21].…”

Section: Introductionmentioning

confidence: 99%

LocalNewton: Reducing Communication Bottleneck for Distributed Learning

Gupta,

Ghosh,

Derezinski

et al. 2021

Preprint

View full text Add to dashboard Cite

To address the communication bottleneck problem in distributed optimization within a master-worker framework, we propose LocalNewton, a distributed second-order algorithm with local averaging. In LocalNewton, the worker machines update their model in every iteration by finding a suitable second-order descent direction using only the data and model stored in their own local memory. We let the workers run multiple such iterations locally and communicate the models to the master node only once every few (say L) iterations. LocalNewton is highly practical since it requires only one hyperparameter, the number L of local iterations. We use novel matrix concentration based techniques to obtain theoretical guarantees for LocalNewton, and we validate them with detailed empirical evaluation. To enhance practicability, we devise an adaptive scheme to choose L, and we show that this reduces the number of local iterations in worker machines between two model synchronizations as the training proceeds, successively refining the model quality at the master. Via extensive experiments using several real-world datasets with AWS Lambda workers and an AWS EC2 master, we show that LocalNewton requires fewer than 60% of the communication rounds (between master and workers) and less than 40% of the end-to-end running time, compared to state-of-the-art algorithms, to reach the same training loss.

show abstract

Straggler Resilient Serverless Computing Based on Polar Codes

Cited by 11 publications

References 14 publications

Polar Coded Computing: The Role of the Scaling Exponent

Polar Coded Computing: The Role of the Scaling Exponent

Computational Polarization: An Information-theoretic Method for Resilient Computing

LocalNewton: Reducing Communication Bottleneck for Distributed Learning

Contact Info

Product

Resources

About