Memory-Limited Execution of Windowed Stream Joins

Abstract

We address the problem of computing approximate answers to continuous sliding-window joins over data streams when the available memory may be insufficient to keep the entire join state. One approximation scenario is to provide a maximum subset of the result, with the objective of losing as few result tuples as possible. An alternative scenario is to provide a random sample of the join result, e.g., if the output of the join is being aggregated. We show formally that neither approximation can be addressed effectively for a slidingwindow join of arbitrary input streams. Previous work has addressed only the maximum-subset problem, and has implicitly used a frequencybased model of stream arrival. We address the sampling problem for this model. More importantly, we point out a broad class of applications for which an age-based model of stream arrival is more appropriate, and we address both approximation scenarios under this new model. Finally, for the case of multiple joins being executed with an overall memory constraint, we provide an algorithm for memory allocation across the joins that optimizes a combined measure of approximation in all scenarios considered. All of our algorithms are implemented and experimental results demonstrate their effectiveness.

Towards Expressive Publish/Subscribe Systems

“…Perhaps the most powerful formal approach is STREAM's CQL query language [25], which extends SQL with support for window queries. Like SQL itself, CQL is declarative and admits of a formal specification [6]; and there are some initial results characterizing a sub-class of queries that can be computed with bounded memory [28,5]. However, as we pointed out in the introduction, it is not clear whether SQL based languages with set semantics are suitable for real-time event detection and composition.…”

Skewed distributions in semi-stream joins: How much can caching help?

“…Load shedding has been discussed for approximate stream join processing [29,30,31], but in these approaches all inputs of the join operator are streams. For semi-stream joins, load shedding has been first discussed in [11].…”

Data-quality Guided Load Shedding for Expensive In-Network Data Processing

“…This model answers the question "which possible candidate m-tuple in the input window will be processed at a given point in time?" The set of objects from which Φ picks its inputs is generally governed by a timeconstraint on the input objects: usually implemented in a time-windowed manner [12]. In this paper, we do not assume any specific windowing scheme.…”

“…In such a case, the node will need to shed (or discard) some of the input data. Most existing work on semantic load shedding [6,12,14] (see Section 5 for the related work) on the other hand tackle the overflow problem in stream databases by aiming to maximize the throughput or at last provide a random sample of the results. In many in-network data processing applications, such as ARIA [11], however, the bottleneck is the processing power of the operators and, in many cases, the throughput is constant.…”