BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning

Abstract

Communication between workers and the master node to collect local stochastic gradients is a key bottleneck in a large-scale federated learning system. Various recent works have proposed to compress the local stochastic gradients to mitigate the communication overhead. However, robustness to malicious attacks is rarely considered in such a setting. In this work, we investigate the problem of Byzantine-robust federated learning with compression, where the attacks from Byzantine workers can be arbitrarily malicious. We point out that a vanilla combination of compressed stochastic gradient descent (SGD) and geometric median-based robust aggregation suffers from both stochastic and compression noise in the presence of Byzantine attacks. In light of this observation, we propose to jointly reduce the stochastic and compression noise so as to improve the Byzantine-robustness. For the stochastic noise, we adopt the stochastic average gradient algorithm (SAGA) to gradually eliminate the inner variations of regular workers. For the compression noise, we apply the gradient difference compression and achieve compression for free. We theoretically prove that the proposed algorithm reaches a neighborhood of the optimal solution at a linear convergence rate, and the asymptotic learning error is in the same order as that of the state-of-the-art uncompressed method. Finally, numerical experiments demonstrate effectiveness of the proposed method. The code is available https://github.com/oyhah/BROADCAST.

Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Top

“…(2) Although the analyses by Wu et al [2020], Zhu and Ling [2021] support inexact geometric median computation, for simplicity of presentation, we assume that geometric median is computed exactly.…”

“…In particular, compressed methods are analyzed in the conjuction with variance reduction [Horváth et al, 2019b, Gorbunov et al, 2020b, Danilova and Gorbunov, 2022, acceleration , Li and Richtárik, 2021, Qian et al, 2021b, decentralized communications [Koloskova et al, 2019, Kovalev et al, 2021, local steps [Basu et al, 2019, Haddadpour et al, 2021, adaptive compression [Faghri et al, 2020], second-order methods , Safaryan et al, 2021, and min-max optimization [Beznosikov et al, 2021[Beznosikov et al, , 2022. However, to our knowledge, only one work studies communication compression in the context of Byzantine-robustness [Zhu and Ling, 2021] (see the discussion above Q2). Our work makes a further step towards closing this significant gap in the literature.…”

“…Perhaps surprisingly, there are not many methods with compressed communication in the context of Byzantine-robust learning. In particular, we are only aware of one work in these settings [Zhu and Ling, 2021], where the authors study Byzantine-robust versions of compressed SGD (BR-CSGD) and SAGA (BR-CSAGA) and also propose a combination of DIANA [Mishchenko et al, 2019, Horváth et al, 2019b with BR-CSAGA called BROADCAST. However, the derived convergence results for these methods have several limitations.…”

“…In addition, it relies on restrictive assumptions. Namely, Zhu and Ling [2021] assume uniform boundedness of the second moment of the stochastic gradient in the analysis of BR-CSGD and BR-CSAGA. This assumption rarely holds in practice, and it also implies the boundedness of the gradients, which contradicts the strong convexity assumption.…”

“…This assumption rarely holds in practice, and it also implies the boundedness of the gradients, which contradicts the strong convexity assumption. Next, although the bounded second-moment assumption is not used in the analysis of BROADCAST, Zhu and Ling [2021] derive the rates of BROADCAST under the assumption that the compression operator is very accurate, which implies that in theory workers apply almost no compression to the communicated messages (see remark (5) under Table 2). Finally, even if there are no Byzantines and no compression, similar to the guarantees for Byrd-SAGA, the rates obtained for BR-CSGD, BR-CSAGA, and BROADCAST are outperformed with a large margin by the known rates for SGD and SAGA.…”