Communication-efficient Distributed Cooperative Learning with Compressed Beliefs

Abstract

We study the problem of distributed cooperative learning, where a group of agents seek to agree on a set of hypotheses that best describes a sequence of private observations. In the scenario where the set of hypotheses is large, we propose a belief update rule where agents share compressed (either sparse or quantized) beliefs with an arbitrary positive compression rate. Our algorithm leverages a unified and straightforward communication rule that enables agents to access wide-ranging compression operators as black-box modules. We prove the almost sure asymptotic exponential convergence of beliefs around the set of optimal hypotheses. Additionally, we show a nonasymptotic, explicit, and linear concentration rate in probability of the beliefs on the optimal hypothesis set. We provide numerical experiments to illustrate the communication benefits of our method. The simulation results show that the number of transmitted bits can be reduced to 5−10% of the non-compressed method in the studied scenarios.

On Arbitrary Compression for Decentralized Consensus and Stochastic Optimization over Directed Networks

“…Check [26,Table 1] for the number of bits required for each operator. Details of other operators can be found in [31], [32].…”

“…We first consider an average consensus problem with d = 300 parameters over a directed Ring graphs with different number of agents n ∈ {20, 50, 100, 200, 500}. We consider a grid over (0, 1] for compression ratio ω, thus top 100ω% [26] as the proper compression operator. We quantify the number of round required for each pair (n, ω), to reach an ε-accuracy where ε=1 • 10 −5 .…”

“…Works in [19]- [22] are proposed to address large communication requirements. More recently, using error-feedback techniques, several efforts have been made to mitigate the communication burden for consensus [23]- [25], inference [26], [27], and optimization problems [24], [28], [29] over undirected graphs.…”

Scalable Average Consensus with Compressed Communications

“…Hereafter, we drop ζ, ω from Q and E for simplicity of notation. The class of randomized operators introduced in (3) embraces a wide range of functions, both sparsification, and quantization, some of which we mention in Example 1 [24].…”

Social Opinion Formation Under Communication Trends

“…Aiming to improve communication efficiency in detection networks, the work [19] proposes communicating with one randomly sampled agent instead of all neighbors at each time instant. Moreover, quantizing the beliefs is possible and studied in [20], [21]. In our work, decreasing the communication burden on the nodes is instead achieved by transmitting partial beliefs.…”