Fault-Tolerant Aggregation for Dynamic Networks

Jesus, Paulo; Baquero, Carlos; Almeida, Paulo Sérgio

doi:10.1109/srds.2010.13

Cited by 19 publications

(26 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approach was inspired by flow-based approaches to averaging to achieve robustness to message loss [7,14]. However, our goal here is not to achieve robustness, but rather to reduce fluctuations in the transmitted values.…”

Section: Flow Compensationmentioning

confidence: 99%

Robust Decentralized Mean Estimation with Limited Communication

Danner

Jelasity

2018

Euro-Par 2018: Parallel Processing

View full text Add to dashboard Cite

Abstract. Mean estimation, also known as average consensus, is an important computational primitive in decentralized systems. When the average of large vectors has to be computed, as in distributed data mining applications, reducing the communication cost becomes a key design goal. One way of reducing communication cost is to add dynamic stateful encoders and decoders to traditional mean estimation protocols. In this approach, each element of a vector message is encoded in a few bits (often only one bit) and decoded by the recipient node. However, due to this encoding and decoding mechanism, these protocols are much more sensitive to benign failure such as message drop and message delay. Properties such as mass conservation are harder to guarantee. Hence, known approaches are formulated under strong assumptions such as reliable communication, atomic non-overlapping transactions or even full synchrony. In this work, we propose a communication efficient algorithm that supports known codecs even if transactions overlap and the nodes are not synchronized. The algorithm is based on push-pull averaging, with novel features to support fault tolerance and compression. As an independent contribution, we also propose a novel codec, called the pivot codec. We demonstrate experimentally that our algorithm improves the performance of existing codecs and the novel pivot codec dominates the competing codecs in the scenarios we studied.

show abstract

Section: Flow Compensationmentioning

confidence: 99%

Robust Decentralized Mean Estimation with Limited Communication

Danner

Jelasity

2018

Euro-Par 2018: Parallel Processing

View full text Add to dashboard Cite

show abstract

“…Alternative gossip-based averaging techniques have been proposed by Eyal et al [5] and Jesus et al [15], to overcome message loss, network churn and topology changes. In LIMOSENSE [5], each node maintains a pair of values, e.g., a weight and an estimation, that is continuously updated during node communication.…”

Section: Related Workmentioning

confidence: 99%

“…In LIMOSENSE [5], each node maintains a pair of values, e.g., a weight and an estimation, that is continuously updated during node communication. Jesus et al [15] propose a technique where each node uses its current set of neighbors and maintains a dynamic mapping of value flows across to them. CROUPIER [16], which is a NAT-aware peer sampling service [2], [3], [17], is another system that uses gossip-based estimation.…”

Section: Related Workmentioning

confidence: 99%

Lightweight gossip-based distribution estimation

Payberah¹,

Kavalionak

Montresor

et al. 2013

2013 IEEE International Conference on Communications (ICC)

View full text Add to dashboard Cite

Monitoring the global state of an overlay network is vital for the self-management of peer-to-peer (P2P) systems. Gossip-based algorithms are a well-known technique that can provide nodes locally with aggregated knowledge about the state of the overlay network. In this paper, we present a gossipbased protocol to estimate the global distribution of attribute values stored across a set of nodes in the system. Our algorithm estimates the distribution both efficiently and accurately. The key contribution of our algorithm is that it has substantially lower overhead than existing distribution estimation algorithms. We evaluated our system in simulation, and compared it against the state-of-the-art solutions. The results show similar accuracy to its counterparts, but with a communication overhead of an order of magnitude lower than them.

show abstract

“…Recent work on this subject [26], [27] based itself on epidemic techniques show that it is possible to obtain accurate estimation of distribution for a given parameter in a scalable and lightweight fashion, which can be used as initial building blocks to our approach. Still, our scenario has particular characteristics that may affect the effectiveness of those protocols, namely a large number of duplicates [27] due to the redundancy, and high churn rates [26], [28] due to large scale, which require further investigation.…”

Section: ) Item Collocationmentioning

confidence: 99%

An epidemic approach to dependable key-value substrates

Matos

Vilaça

Pereira

et al. 2011

2011 IEEE/IFIP 41st International Conference on Dependable Systems and Networks Workshops (DSN-W)

View full text Add to dashboard Cite

Abstract-The sheer volumes of data handled by today's Internet services demand uncompromising scalability from the persistence substrates. Such demands have been successfully addressed by highly decentralized key-value stores invariably governed by a distributed hash table. The availability of these structured overlays rests on the assumption of a moderately stable environment. However, as scale grows with unprecedented numbers of nodes the occurrence of faults and churn becomes the norm rather than the exception, precluding the adoption of rigid control over the network's organization.In this position paper we outline the major ideas of a novel architecture designed to handle today's very large scale demand and its inherent dynamism. The approach rests on the well-known reliability and scalability properties of epidemic protocols to minimize the impact of churn. We identify several challenges that such an approach implies and speculate on possible solutions to ensure data availability and adequate access performance.

show abstract

Fault-Tolerant Aggregation for Dynamic Networks

Cited by 19 publications

References 16 publications

Robust Decentralized Mean Estimation with Limited Communication

Robust Decentralized Mean Estimation with Limited Communication

Lightweight gossip-based distribution estimation

An epidemic approach to dependable key-value substrates

Contact Info

Product

Resources

About