Generating Reproducible Out-of-Order Data Streams

Grulich, Philipp M.; Traub, Jonas; Breß, Sebastian; Katsifodimos, Asterios; Markl, Volker; Rabl, Tilmann

doi:10.1145/3328905.3332511

Cited by 10 publications

(4 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Consequently, the binary tree is updated by purging the expired p-value. The query result is computed by aggregating the t-value (16) and the g-value (15), which is 16.…”

Section: T-value = Updatet Ree(expv Alue)mentioning

confidence: 99%

See 1 more Smart Citation

CPiX: Real-Time Analytics Over Out-of-Order Data Streams by Incremental Sliding-Window Aggregation

Bou

Kitagawa

Amagasa

2022

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Stream processing is used in various fields. In the field of big data, stream aggregation is a popular processing technique, but it suffers serious setbacks when the order of events (e.g., stream elements) occurring is different from the order of events arriving to the systems. Such data streams are called "non-FIFO steams". This phenomenon usually occurs in a distributed environment due to many factors, such as network disruptions, delays, etc. Many analyzing scenarios require efficient processing of such non-FIFO streams to meet various data processing requirements. This paper proposes an efficient scalable checkpoint-based bidirectional indexing approach, called CP iX, for faster real-time analysis over non-FIFO streams. CPiX maintains the partial aggregation results in an on-demand manner per checkpoint. CPiX needs less time and space than the state-of-the-art approach. Extensive experiments confirm that CPiX can deal with out-of-order streams very efficiently and is, on average, about 3.8 times faster than the state-of-the-art approach while consuming less memory.

show abstract

“…Consequently, the binary tree is updated by purging the expired p-value. The query result is computed by aggregating the t-value (16) and the g-value (15), which is 16.…”

Section: T-value = Updatet Ree(expv Alue)mentioning

confidence: 99%

“…One practical issue is how to handle an infinite sequence of streaming records [5], [6], [7], [8], [9], [15] using finite memory (or computational resources). Using a window to extract partial data from an infinite stream is one solution as a window limits the evaluation scope given an operation over a sequence of records.…”

Section: Introductionmentioning

confidence: 99%

CPiX: Real-Time Analytics Over Out-of-Order Data Streams by Incremental Sliding-Window Aggregation

Bou

Kitagawa

Amagasa

2022

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

“…The open-source stream generator proposed by Grulich et al [27] enables the evaluation of modern stream processing systems. It produces deterministic data streams from arbitrary input data sets with different data rates, distributions, and characteristics such as the fraction of out-of-order tuples and their delay.…”

Section: Applicationsmentioning

confidence: 99%

“…Condor [48], Stream Generator [27], I2 [64], M4 [32] speed networks, to accelerate the performance of stream processing systems. Sect.…”

Section: Introductionmentioning

confidence: 99%

Algorithms for Windowed Aggregations and Joins on Distributed Stream Processing Systems

et al. 2022

Self Cite

View full text Add to dashboard Cite

Window aggregations and windowed joins are central operators of modern real-time analytic workloads and significantly impact the performance of stream processing systems.This paper gives an overview of state-of-the-art research in this area conducted by the Berlin Institute for the Foundations of Learning and Data (BIFOLD) and the Technische Universität Berlin. To this end, we present different algorithms for efficiently processing windowed operators and discuss techniques for distributed stream processing. Recently, several approaches have leveraged modern hardware for windowed stream processing, which we will also include in this overview. Additionally, we describe the integration of windowed operators into various stream processing systems and diverse applications that use specialized window operations.

show abstract