DATAFLASKS: Epidemic Store for Massive Scale Systems

Maia, Francisco; Matos, Miguel; Vilaça, Ricardo; Pereira, José; Oliveira, Rui; Rivière, Étienne

doi:10.1109/srds.2014.34

Cited by 6 publications

(13 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead, we focus on analyzing storage and network savings achievable by our system. Similarly, the validation of DDFLASKS scalability to thousands of nodes and resiliency to high churn rations is already addressed in previous work [18].…”

Section: Discussionmentioning

confidence: 96%

“…The pivotal idea guiding the design of DATAFLASKS is decentralization, where each node is autonomous and all nodes play the same role [18]. A node progresses relying solely on local decisions without depending on any other node and on any kind of hierarchy.…”

Section: Dataflasks: Epidemic Store For Massive Scale Systemsmentioning

confidence: 99%

“…Then, the node places himself on one of those groups guaranteeing that system nodes are uniformly distributed across the different groups. For a detailed description of the protocol please refer to [18]. Once in a group, each time a put operation is issued for a certain key, that key is mapped deterministically to a group by using an hash function.…”

Section: Dataflasks: Epidemic Store For Massive Scale Systemsmentioning

confidence: 99%

“…Recent research work proposed a data store entirely built with epidemic protocols, tailored precisely for large scale environments [18]. The success of DATAFLASKS, with respect to coping with high levels of system dynamism, lies in its autonomous and unstructured approach to node organization and in its pro-active approach to fault tolerance.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

$${\textsc {DDFlasks}}$$: Deduplicated Very Large Scale Data Store

Maia

Paulo

Coelho

et al. 2017

Distributed Applications and Interoperable Systems

Self Cite

View full text Add to dashboard Cite

With the increasing number of connected devices, it becomes essential to find novel data management solutions that can leverage their computational and storage capabilities. However, developing very large scale data management systems requires tackling a number of interesting distributed systems challenges, namely continuous failures and high levels of node churn. In this context, epidemic-based protocols proved suitable and effective and have been successfully used to build DATAFLASKS, an epidemic data store for massive scale systems. Ensuring resiliency in this data store comes with a significant cost in storage resources and network bandwidth consumption. Deduplication has proven to be an efficient technique to reduce both costs but, applying it to a largescale distributed storage system is not a trivial task. In fact, achieving significant space-savings without compromising the resiliency and decentralized design of these storage systems is a relevant research challenge. In this paper, we extend DATAFLASKS with deduplication to design DDFLASKS. This system is evaluated in a real world scenario using Wikipedia snapshots, and the results are twofold. We show that deduplication is able to decrease storage consumption up to 63% and decrease network bandwidth consumption by up to 20%, while maintaining a fully-decentralized and resilient design.

show abstract

Section: Discussionmentioning

confidence: 96%

Section: Dataflasks: Epidemic Store For Massive Scale Systemsmentioning

confidence: 99%

Section: Dataflasks: Epidemic Store For Massive Scale Systemsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

$${\textsc {DDFlasks}}$$: Deduplicated Very Large Scale Data Store

Maia

Paulo

Coelho

et al. 2017

Distributed Applications and Interoperable Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…These services can be smaller applications or protocols that are used as building blocks for a larger application. For instance, in a typical epidemic application [9], different protocols are used as they rely on each other. In our platform, each service is implemented in a JSR-223 compliant language and implementing the exposed API.…”

Section: Framework Designmentioning

confidence: 99%

Practical Evaluation of Large Scale Applications

Jorge

Maia

Matos

et al. 2015

Distributed Applications and Interoperable Systems

Self Cite

View full text Add to dashboard Cite

Abstract. Designing and implementing distributed systems is a hard endeavor, both at an abstract level when designing the system, and at a concrete level when implementing, debugging and evaluating it. This stems not only from the inherent complexity of writing and reasoning about distributed software, but also from the lack of tools for testing and evaluating it under realistic conditions. Moreover, the gap between the protocols' specifications found on research papers and their implementations on real code is huge, leading to inconsistencies that often result in the implementation no longer following the specification. As an example, the specification of the popular Chord DHT comprises a few dozens of lines, while its Java implementation, OpenChord, is close to twenty thousand lines, excluding libraries. This makes it hard and error prone to change the implementation to reflect changes in the specification, regardless of programmers' skill. Besides, critical behavior due to the unpredictable interleaving of operations and network uncertainty, can only be observed on a realistic setting, limiting the usefulness of simulation tools. We believe that being able to write an algorithm implementation very close to its specification, and evaluating it in a real environment is a big step in the direction of building better distributed systems. Our approach leverages the MINHA platform to offer a set of built in primitives that allows one to program very close to pseudo-code. This high level implementation can interact with off-the-shelf existing middleware and can be gradually replaced by a production-ready Java implementation. In this paper, we present the system design and showcase it using a well-known algorithm from the literature.

show abstract

Totally Ordered Replication for Massive Scale Key-Value Stores

Ribeiro¹,

Machado²,

Maia³

et al. 2018

Distributed Applications and Interoperable Systems

Self Cite

View full text Add to dashboard Cite

Scalability is one of the most relevant features of today's data management systems. In order to achieve high scalability and availability, recent distributed key-value stores refrain from costly replica coordination when processing requests. However, these systems typically do not perform well under churn. In this paper, we propose DataFlagons, a large-scale key-value store that integrates epidemic dissemination with a probabilistic total order broadcast algorithm. By ensuring that all replicas process requests in the same order, DataFlagons provides probabilistic strong data consistency while achieving high scalability and robustness under churn.

show abstract

DATAFLASKS: Epidemic Store for Massive Scale Systems

Cited by 6 publications

References 35 publications

$${\textsc {DDFlasks}}$$: Deduplicated Very Large Scale Data Store

$${\textsc {DDFlasks}}$$: Deduplicated Very Large Scale Data Store

Practical Evaluation of Large Scale Applications

Totally Ordered Replication for Massive Scale Key-Value Stores

Contact Info

Product

Resources

About