Quality-Driven Continuous Query Execution over Out-of-Order Data Streams

Abstract

Executing continuous queries over out-of-order data streams, where tuples are not ordered according to timestamps, is challenging; because high result accuracy and low result latency are two conflicting performance metrics. Although many applications allow trading exact query results for lower latency, they still expect the produced results to meet a certain quality requirement. However, none of existing disorder handling approaches have considered minimizing the result latency while meeting user-specified requirements on the quality of query results.In this demonstration, we showcase AQ-K-slack, an adaptive, buffer-based disorder handling approach, which supports executing sliding window aggregate queries over outof-order data streams in a quality-driven manner. By adapting techniques from the field of sampling-based approximate query processing and control theory, AQ-K-slack dynamically adjusts the input buffer size at query runtime to minimize the result latency, while respecting a user-specified threshold on relative errors in produced query results.We demonstrate a prototype stream processing system, which extends SAP Event Stream Processor with the implementation of AQ-K-slack. Through an interactive interface, the audience will learn the effect of different factors, such as the aggregate function, the window specification, the result error threshold, and stream properties, on the latency and the accuracy of query results. Moreover, they can experience the effectiveness of AQ-K-slack in obtaining user-desired latency vs. result accuracy trade-offs, compared to naive disorder handling approaches that make extreme trade-offs. For instance, by scarifying 1% result accuracy, our system can reduce the result latency by 80% when compared to the state of the art.

Dynamic Buffer Sizing for Out-of-order Event Compensation for Time-sensitive Applications

“…An Adaptive, Quality-drive K-slack (AQ-K-slack) is another K-Slack based approach that is shown along the publications [12,13] in which Ji et al are able to reduce the latency more than 50% compared to the standard K-slack approach. In contrast to our approach that is generally applicable and independent of the successive computation logic in which the distribution of the transmission delays are investigated to calculate the buffer size, the authors introduce an error model for query results in which the window coverage metric is bounded to the windowing function.…”

Khronos

“…Significant research efforts focus on the management of late event arrivals in the context of Complex Event Processing systems, such as punctuation [24,27], speculation [15,18] and buffer-based data-structures [9][10][11]29]. However, these approaches often rely on the user (e.g.…”

GeneaLog

“…In a query, source tuples are delivered by Sources, analyzed by a Directed Acyclic Graph (DAG) of operators which can also produce new tuples (as described later in this section) and, eventually, delivered as sink tuples to Sinks. When the tuples of each source stream are fed to the operators of a query in timestamp order (either because Sources deliver timestamp-sorted streams as in [6,18,26] or by leveraging sorting techniques such as [25]) and each operator produces timestampsorted output streams (merging in timestamp order its input tuples if the latter are delivered by multiple input streams, as discussed in [18][19][20]35]) a query's execution is deterministic. In a nutshell, this is given by the fact that each processing step depends on the notion of time carried by the tuples themselves (attribute ts) and is affected neither by the latency incurred in transmitting tuples from an operator to another operator nor by the interleaving of tuples to an operator with multiple input streams.…”