Scalable and adaptive online joins

Abstract

Scalable join processing in a parallel shared-nothing environment requires a partitioning policy that evenly distributes the processing load while minimizing the size of state maintained and number of messages communicated. Previous research proposes static partitioning schemes that require statistics beforehand. In an online or streaming environment in which no statistics about the workload are known, traditional static approaches perform poorly.This paper presents a novel parallel online dataflow join operator that supports arbitrary join predicates. The proposed operator continuously adjusts itself to the data dynamics through adaptive dataflow routing and state repartitioning. The operator is resilient to data skew, maintains high throughput rates, avoids blocking behavior during state repartitioning, takes an eventual consistency approach for maintaining its local state, and behaves strongly consistently as a black-box dataflow operator. We prove that the operator ensures a constant competitive ratio 3.75 in data distribution optimality and that the cost of processing an input tuple is amortized constant, taking into account adaptivity costs. Our evaluation demonstrates that our operator outperforms the state-of-the-art static partitioning schemes in resource utilization, throughput, and execution time.

Bleach: A Distributed Stream Data Cleaning System

“…In contrast, stream processing requires to be real-time, a challenge that has drawn increasing attention from researchers [2,14,21,11]. Nevertheless, stream data cleaning approaches are still in their infancy.…”

Load balancing and skew resilience for parallel joins

“…We refer to a set of cells (that is, the corresponding input tuples) assigned to a single machine for local processing as a region. We adhere to rectangular regions, as opposed to rectilinear or non-contiguous regions, to incur minimal storage and communication costs [9].…”

“…The content-insensitive partitioning scheme, CI (called 1-Bucket in [4], [9]), illustrated in Figure 1b, assigns all cells (n 2 of them) to machines, regardless of the join condition. Thus, regions cover the entire join matrix.…”

Incorporating change detection in the monitoring phase of adaptive query processing

“…To overcome this problem, a plethora of Adaptive Query Processing (AQP) techniques have been recently proposed in the literature aiming to adapt the runtime query plan in respond to changes in the execution environment or the characteristics of the streaming data [4][5][6][7][8]. The rationale followed by these AQP techniques can be condensed into a three-phase procedure, called adaptivity loop [9].…”

“…First, the Adjust feedback function is called after a change is detected, instead of Initialize (line 24). Second, if the query plan is reoptimized, the monitoring phase calls the Initialize function of the change detection algorithm, so as the change detection algorithm to entirely forget its runtime state (lines [5][6][7][8]. Recall that after the Initialize state, a "feedback-full" algorithm must collect feedback prior to be operational.…”