A survey of state management in big data processing systems

To, Quoc-Cuong; Soto, Juan; Markl, Volker

doi:10.1007/s00778-018-0514-9

Cited by 64 publications

(36 citation statements)

References 96 publications

(216 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While scaling‐in/out stateless operators can be achieved by just turning off/on operator replicas, elasticity of stateful operators requires state migration and repartitioning among the replicas, because the system needs to preserve the consistency of the operations . State management in DPS systems is nicely surveyed by To et al; in the following, we focus on the issues that are more closely related to our work.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Optimal operator deployment and replication for elastic distributed data stream processing

Cardellini

Presti

Nardelli

et al. 2017

Concurrency and Computation

View full text Add to dashboard Cite

Summary Processing data in a timely manner, data stream processing (DSP) applications are receiving an increasing interest for building new pervasive services. Due to the unpredictability of data sources, these applications often operate in dynamic environments; therefore, they require the ability to elastically scale in response to workload variations. In this paper, we deal with a key problem for the effective runtime management of a DSP application in geo‐distributed environments: We investigate the placement and replication decisions while considering the application and resource heterogeneity and the migration overhead, so to select the optimal adaptation strategy that can minimize migration costs while satisfying the application quality of service (QoS) requirements. We present elastic DSP replication and placement (EDRP), a unified framework for the QoS‐aware initial deployment and runtime elasticity management of DSP applications. In EDRP, the deployment and runtime decisions are driven by the solution of a suitable integer linear programming problem, whose objective function captures the relative importance between QoS goals and reconfiguration costs. We also present the implementation of EDRP and the related mechanisms on Apache Storm. We conduct a thorough experimental evaluation, both numerical and prototype‐based, that shows the benefits achieved by EDRP on the application performance.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Madsen et al 24 28 ; in the following, we focus on the issues that are more closely related to our work.…”

Section: Related Workmentioning

confidence: 99%

Optimal operator deployment and replication for elastic distributed data stream processing

Cardellini

Presti

Nardelli

et al. 2017

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…The main distinguishing feature of the existing elasticity policies regards their being centralized, fully decentralized, or hybrid. Another characterizing aspect is related to taking into account or not the reconfiguration costs that arise after a scaling-in/out decision as well as a migration from one computing resource to another and are particularly burdensome in case of stateful operators [31].…”

Section: Elasticity Policiesmentioning

confidence: 99%

Multi-Level Elasticity for Wide-Area Data Streaming Systems: A Reinforcement Learning Approach

et al. 2018

View full text Add to dashboard Cite

The capability of efficiently processing the data streams emitted by nowadays ubiquitous sensing devices enables the development of new intelligent services. Data Stream Processing (DSP) applications allow for processing huge volumes of data in near real-time. To keep up with the high volume and velocity of data, these applications can elastically scale their execution on multiple computing resources to process the incoming data flow in parallel. Being that data sources and consumers are usually located at the network edges, nowadays the presence of geo-distributed computing resources represents an attractive environment for DSP. However, controlling the applications and the processing infrastructure in such wide-area environments represents a significant challenge. In this paper, we present a hierarchical solution for the autonomous control of elastic DSP applications and infrastructures. It consists of a two-layered hierarchical solution, where centralized components coordinate subordinated distributed managers, which, in turn, locally control the elastic adaptation of the application components and deployment regions. Exploiting this framework, we design several self-adaptation policies, including reinforcement learning based solutions. We show the benefits of the presented self-adaptation policies with respect to static provisioning solutions, and discuss the strengths of reinforcement learning based approaches, which learn from experience how to optimize the application performance and resource allocation.

show abstract

“…Deletes in LSM-trees. LSM-trees are employed as the storage layer for relational systems [28], streaming systems [2,41,65], and pure key-value storage [52,68]. As a result, an LSM delete operation may be triggered by various logical operations, not limited to user-driven deletes.…”

Section: Introductionmentioning

confidence: 99%

Lethe: A Tunable Delete-Aware LSM Engine

Sarkar

Papon

Staratzis

et al. 2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Data-intensive applications fueled the evolution of log structured merge (LSM) based key-value engines that employ the out-of-place paradigm to support high ingestion rates with low read/write interference. These benefits, however, come at the cost of treating deletes as a second-class citizen. A delete inserts a tombstone that invalidates older instances of the deleted key. State-of-the-art LSM engines do not provide guarantees as to how fast a tombstone will propagate to persist the deletion. Further, LSM engines only support deletion on the sort key. To delete on another attribute (e.g., timestamp), the entire tree is read and re-written. We highlight that fast persistent deletion without affecting read performance is key to support: (i) streaming systems operating on a window of data, (ii) privacy with latency guarantees on the right-to-be-forgotten, and (iii) en masse cloud deployment of data systems that makes storage a precious resource.To address these challenges, in this paper, we build a new key-value storage engine, Lethe, that uses a very small amount of additional metadata, a set of new delete-aware compaction policies, and a new physical data layout that weaves the sort and the delete key order. We show that Lethe supports any user-defined threshold for the delete persistence latency offering higher read throughput (1.17 − 1.4×) and lower space amplification (2.1 − 9.8×), with a modest increase in write amplification (between 4% and 25%). In addition, Lethe supports efficient range deletes on a secondary delete key by dropping entire data pages without sacrificing read performance nor employing a costly full tree merge.

show abstract

A survey of state management in big data processing systems

Cited by 64 publications

References 96 publications

Optimal operator deployment and replication for elastic distributed data stream processing

Optimal operator deployment and replication for elastic distributed data stream processing

Multi-Level Elasticity for Wide-Area Data Streaming Systems: A Reinforcement Learning Approach

Lethe: A Tunable Delete-Aware LSM Engine

Contact Info

Product

Resources

About