DRIVE: One-bit Distributed Mean Estimation

EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning

“…. DRIVE shows consistent improvement over the state of the art, over a collection of distributed and federated learning tasks [1]. This technical report generalizes DRIVE to any b ą 0 bandwidth constraint and discusses additional extensions (e.g., resiliency to arbitrary packet drops).…”

“…We assume that each client has access to randomness that is shared with the PS. This assumption is standard (e.g., is used in [1,7,11]) and can be implemented by having a shared seed (e.g., the combination of client ID and epoch number) for a pseudo-random number generator. For example, a client can generate a random rotation matrix that is also available to the PS as they use the same random bits.…”

“…Therefore, applying stochastic quantization to the Kashin coefficients allows an NMSE of Op 1 n q using Opdq bits. Compared with the Hadamard transform of [7], using Kashin's representation yields a lower NMSE at the cost of additional computation [1,8].…”

“…DRIVE [1] is a recently introduced algorithm for the p1 `op1qq-DME problem (i.e., with one bit per coordinate and some lower-order overhead). To the best of our knowledge, DRIVE is the first algorithm with Algorithm 1 DRIVE [1] Client c: 1: Compute R c px c q, S c .…”

“…While a bit budget of one bit per coordinate was thoroughly studied [1,11,12,13,14,15], and is applied to accelerate distributed learning systems [6,16], our approach allows one to use DRIVE with b ą 1 for deployment on fast networks or b ă 1 (e.g., half a bit per coordinate) when the network is the bottleneck.…”