Concurrent data structures for efficient streaming aggregation

Abstract

We briefly describe our study on the problem of streaming multiway aggregation [5], where large data volumes are received from multiple input streams. Multiway aggregation is a fundamental computational component in data stream management systems, requiring low-latency and high throughput solutions. We focus on the problem of designing concurrent data structures enabling for low-latency and highthroughput multiway aggregation; an issue that has been overlooked in the literature. We propose two new concurrent data structures and their lock-free linearizable implementations, supporting both order-sensitive and order-insensitive aggregate functions. Results from an extensive evaluation show significant improvement in the aggregation performance, in terms of both processing throughput and latency over the commonly-used techniques based on queues.

A Systematic Methodology for Optimization of Applications Utilizing Concurrent Data Structures

“…We notice that the latter implementation provides the highest throughput, along with the highest power consumption (42 and 5.1 percent higher than S1 respectively). Similar results related to the efficiency in terms of throughput of lock-free data structures for streaming aggregation implementations can be found in the literature [21]. Fig.…”

“…Dedup is a data deduplication algorithm taken from the Parsec Benchmark Suite [20]. Finally, the last application performs a multiway data Streaming Aggregation [21]. We used this application in order to demonstrate the implementation of the methodology in applications that utilize more than one concurrent data structure.…”

“…Streaming Aggregation [21] key-value pairs sorted storage the latter, the threads are often suspended by the kernel, while they wait for the lock to be released. There are two valid implementations for the Patricia benchmark for the I.MX 6 chip.…”

Power‐aware pipelining with automatic concurrency control

“…The data structure allows a set of readers to consume ready tuples based on their timestamp by guaranteeing deterministic processing. The same idea has been recently extended to support multiway aggregation operators in the work of Cederman et al, showing high‐throughput and low‐latency levels with respect to traditional implementation techniques.…”

Deterministic, elastic and real - time processing in the big data era

“…A stream operator's implementation is considered deterministic, if, given the same sequences of input tuples, the same sequence of output tuples will be produced, independently of the tuples' inter-arrival time. In [60] the ScaleGate data structure guarantees that data arriving from different input streaming sources are processed in the correct order by the join operator, while it has also been used for scalable streaming aggregates [61] and analytics [34]. ScaleGate stalls the processing of each tuple until it is certain that no other Queueing Threshold=1000 Queueing Threshold=5000 Queueing Threshold=10000 Queueing Threshold=20000 Fig.…”