Robust Decentralized Mean Estimation with Limited Communication

Abstract

Abstract. Mean estimation, also known as average consensus, is an important computational primitive in decentralized systems. When the average of large vectors has to be computed, as in distributed data mining applications, reducing the communication cost becomes a key design goal. One way of reducing communication cost is to add dynamic stateful encoders and decoders to traditional mean estimation protocols. In this approach, each element of a vector message is encoded in a few bits (often only one bit) and decoded by the recipient node. However, due to this encoding and decoding mechanism, these protocols are much more sensitive to benign failure such as message drop and message delay. Properties such as mass conservation are harder to guarantee. Hence, known approaches are formulated under strong assumptions such as reliable communication, atomic non-overlapping transactions or even full synchrony. In this work, we propose a communication efficient algorithm that supports known codecs even if transactions overlap and the nodes are not synchronized. The algorithm is based on push-pull averaging, with novel features to support fault tolerance and compression. As an independent contribution, we also propose a novel codec, called the pivot codec. We demonstrate experimentally that our algorithm improves the performance of existing codecs and the novel pivot codec dominates the competing codecs in the scenarios we studied.

Gossip Learning as a Decentralized Alternative to Federated Learning

“…As for future work, the most promising direction is the design and evaluation of more sophisticated compression techniques [5] for both federated and gossip learning. Also, in both cases, there is a lot of opportunity to optimize the communication pattern by introducing asynchrony to federated learning, or adding flow control to gossip learning [6].…”

FoBSim: An extensible open-source simulation tool for integrated Fog-Blockchain systems

“…Although the implementation of such protocol is relatively simple, it is differently implemented in different systems. Some famous examples of efficient gossiping protocols include the Push-Sum protocol [69], the Push-Flow algorithm [70], and different versions of the Push-Pull averaging protocol [71]. Furthermore, we found that its application in FoBSim was useful, when the PoW CA is used in a multiprocessing scenario, with a relatively low puzzle difficulty.…”

Decentralized machine learning using compressed push-pull averaging

“…Our compressed push-pull learning algorithm is based on a compressed push-pull averaging protocol that compresses communication using codecs [7]. The nodes perodically train their model on the local data, as well as perform distributed averaging of the models.…”

“…The techniques used for compression and ensuring sum preservation are largly unchanged from [7] and we do not go into great detail concerning these. One difference worth mentioning, though, is that we omitted the flow compensation component, because it had a negative influence on the machine learning performance.…”

“…Our contribution in this paper is twofold. First, we adapt the compressed push-pull averaging algorithm from [7] for gossip learning, and achieve a higher communication efficiency than previous methods based on subsampling. Second, we evaluate the solution over datasets including a transfer learning dataset, thereby demonstrating that it is feasible to adapt pre-existing deep neural network models to another domain by training only their last layer, which makes them accessible for gossip learning applications.…”