Load Shedding for Window Joins on Multiple Data Streams

Abstract

We consider the problem of semantic load shedding for continuous queries containing window joins on multiple data streams and propose a robust approach that is effective with the different semantic accuracy criteria that are required in different applications. In fact, our approach can be used to (i) maximize the number of output tuples produced by joins, and (ii) optimize the accuracy of complex aggregates estimates under uniform random sampling. We first consider the problem of computing maximal subsets of approximate window joins over multiple data streams. Previously proposed approaches are based on multiple pairwise joins and, in their load-shedding decisions, disregard the content of streams outside the joined pairs. To overcome these limitations, we optimize our load-shedding policy using various predictors of the productivity of each tuple in the window. To minimize processing costs, we use a fastand-light sketching technique to estimate the productivity of the tuples. We then show that our method can be generalized to produce statistically accurate samples, as needed in, e.g., the computation of averages, quantiles, and stream mining queries. Tests performed on both synthetic and reallife data demonstrate that our method outperforms previous approaches, while requiring comparable amounts of time and space.

Optimal load shedding with aggregates and mining queries

“…Thus, in the context of frequent pattern mining, pattern verification [18], and any other application that consists of only counting queries, we make the following observation: from (14) and (15) we notice that both the uniform and proportional 16 load shedding policies produce the same total variance (G uni = G prop ). However, the uniform approach is still more favorable since it does not require knowing the f k values, while the proportional method does.…”

“…The prior work has addressed the processing of join queries under load shedding [10], [11], [16], which usually involves adhoc heuristics. For aggregate queries, which is the focus of this paper, we instead use random load shedding.…”

A Review of Window Query Processing for Data Streams

“…They considered the MAX-subset measure, which maximizes the number of tuples in the approximate output of the join. The MAX-subset measure was considered for load shedding in many algorithms [40][41][42]. Das et al [26] also proposed two heuristics to determine the priority of tuples in an online join: PROB and LIFE.…”

“…Their method focused on the memory-limited situation. Many other studies including [26,[40][41][42] also assumed the memory-limited situation, and then discussed their load shedding algorithms. On the other hand, Gedik et al [39,44] emphasized a situation where the CPU becomes a bottleneck (i.e., when an input arrival rate exceeds CPU processing speed), and then proposed load shedding techniques to shed the CPU load.…”

“…• Frequency-based model [26,[39][40][41][42]: The priority of a tuple s is determined in proportion to the number of tuples in windows, whose value is the same as that of s.…”