Approximate Gradient Coding via Sparse Random Graphs

Abstract

Distributed algorithms are often beset by the straggler effect, where the slowest compute nodes in the system dictate the overall running time. Coding-theoretic techniques have been recently proposed to mitigate stragglers via algorithmic redundancy. Prior work in coded computation and gradient coding has mainly focused on exact recovery of the desired output. However, slightly inexact solutions can be acceptable in applications that are robust to noise, such as model training via gradient-based algorithms. In this work, we present computationally simple gradient codes based on sparse graphs that guarantee fast and approximately accurate distributed computation. We demonstrate that sacrificing a small amount of accuracy can significantly increase algorithmic robustness to stragglers.

Approximate Gradient Coding for Heterogeneous Nodes

“…• We provide a thorough analysis for these two schemes under shuffling and obtain expressions for their expected optimal decoding error. Our analysis for the FRC scheme extends the existing results [8,Thm. 6] for the heterogeneous straggler model.…”

“…Let A ∈ R n×r be the submatrix is formed by considering the columns of B that correspond to non-stragglers. This matrix A is termed as a non-straggler matrix [8]. For an (s − 1)-tolerant coding scheme, master is guaranteed to compute the exact gradient g when r ≥ n − s + 1 [2].…”

“…The average-case model corresponds to the case where the number of actual stragglers can be more than s max and the master might not always be able to compute the exact value of the gradient. In this case, the master computes an approximate gradient using the computations performed by the set of non-stragglers [8]- [12]. In many applications, the average-case straggler model is desirable as s max might not be known a priori.…”

A Survey on Fault-tolerance in Distributed Optimization and Machine Learning

“…One of the applications of coding theory is to enable error correction [50]. Gradient coding is originally proposed as a straggler mitigation method [106], which is used to speed up synchronous distributed first-order methods [17,23,89]. Several works build upon it and extend it to the adversarial setup.…”

“…The major difference between Algorithms 2 and DGD is (17): instead of updating the estimates by the sum of all agents updates, BGD uses an aggregation rule GradFilter (•). Generally speaking, GradFilter : R d × n → R d is a function that takes n vectors of d-dimension, and output a vector of d-dimension.…”