Quality-driven processing of sliding window aggregates over out-of-order data streams

Abstract

One fundamental challenge in data stream processing is to cope with the ubiquity of disorder of tuples within a stream caused by network latency, operator parallelization, merging of asynchronous streams, etc. High result accuracy and low result latency are two conflicting goals in out-of-order stream processing. Different applications may prefer different extent of trade-offs between the two goals. However, existing disorder handling solutions either try to meet one goal to the extreme by sacrificing the other, or try to meet both goals but have shortcomings including unguaranteed result accuracy or increased complexity in operator implementation and application logic.To meet different application requirements on the latency versus result accuracy trade-off in out-of-order stream processing, in this paper, we propose to make this trade-off user-configurable. Particularly, focusing on sliding window aggregates, we introduce AQ-K-slack, a buffer-based qualitydriven disorder handling approach. AQ-K-slack leverages techniques from the fields of sampling-based approximate query processing and control theory. It can adjust the input buffer size dynamically to minimize the result latency, while respecting user-specified threshold on relative errors in produced query results. AQ-K-slack requires no a priori knowledge of disorder characteristics of data streams, and imposes no changes to the query operator implementation or the application logic. Experiments over real-world out-of-order data streams show that, compared to the stateof-art, AQ-K-slack can reduce the average buffer size, thus the average result latency, by at least 51% while respecting user-specified requirement on the accuracy of query results.

Optimal and general out-of-order sliding-window aggregation

“…1 fun query(t from : Time, t to : Time) : Agg 2 node from , node to ← searchNode(t from ), searchNode(t to ) 3 node top ← leastCommonAncestor(node from , node to ) 4 return queryRec(node top , t from , t to ) t next ≤ t to ? +∞ : t to ) 16 for i ∈ [0, ..., node.arity -2] 17 t i ← node.getTime(i) 18 if t from ≤ t i and t i ≤ t to 19 res ← res ⊗ node.getValue(i) 20 if not node.isLeaf() and i + 1 < node.arity − 2 26 if not node.isLeaf() 27 t curr ← node.getTime(node.arity -2)…”

“…This is also expected, since in the case of d = 0, the fingers enable amortized constant updates. When the aggregation operator is expensive, the 2 1 2 3 2 5 2 7 2 9 2 11 2 13 2 15 2 17 2 19 finger B-trees have significantly lower latency, because they have to repair fewer partial aggregates.…”

“…One can reduce the delay by optimistically performing computation over transactional memory [11] and performing commits in-order. Finally, one can tune the trade-off between quality and latency by adaptively adjusting buffer sizes [19]. In contrast to buffering approaches, FiBA can handle arbitrary lateness without sacrificing quality nor significant latency.…”

Dynamic Buffer Sizing for Out-of-order Event Compensation for Time-sensitive Applications

“…An Adaptive, Quality-drive K-slack (AQ-K-slack) is another K-Slack based approach that is shown along the publications [12,13] in which Ji et al are able to reduce the latency more than 50% compared to the standard K-slack approach. In contrast to our approach that is generally applicable and independent of the successive computation logic in which the distribution of the transmission delays are investigated to calculate the buffer size, the authors introduce an error model for query results in which the window coverage metric is bounded to the windowing function.…”

Harnessing sliding-window execution semantics for parallel stream processing

“…Instead, a stream is out-of-order if it may be possible that τ (t i ) < τ (t j ) with i > j and tuple t i is called a late arrival. Disordered streams exist in the real practice and require proper mechanisms to be processed correctly (e.g., punctuations, reordering buffering [16,17]). In the rest of this paper we will focus on ordered streams only.…”