DHTJoin: processing continuous join queries using DHT networks

Abstract

Continuous query processing in data stream management systems (DSMS) has received considerable attention recently. Many applications share the same need for processing data streams in a continuous fashion. For most distributed streaming applications, the centralized processing of continuous queries over distributed data is simply not viable. This paper addresses the problem of computing approximate answers to continuous join queries over distributed data streams. We present a new method, called DHTJoin, which combines hash-based placement of tuples in a Distributed Hash Table (DHT) and dissemination of queries by exploiting the embedded trees in the underlying DHT, thereby incurring little overhead. DHTJoin also deals with join attribute value skew which may hurt load balancing and result completeness. We provide a performance evaluation of DHTJoin which shows that it can achieve significant performance gains in terms of network traffic.

HYBRIDJOIN for Near-Real-Time Data Warehousing

“…This real-time processing of the update stream introduces the interesting challenges related to throughput for join algorithms. Some techniques have been introduced already to process join queries over continuous streaming data (Golab & Özsu, 2003) (Babu & Widom, 2001) (Hammad, Aref, & Elmagarmid, 2008) (Palma, Akbarinia, Pacitti, & Valduriez, 2009) (Kim & Park, 2005) (Nguyen, Brezany, Tjoa, & Weippl, 2005). In this section we will outline the well known work that has already been done in this area with a particular focus on those which are closely related to our problem domain.…”

StreamCloud: A Large Scale Data Streaming System

“…This operator is parallelized as follows (see Figure 3 lines 11-23). Given a subcluster of N nodes to execute the CP operator, each tuple is sent to M = √ N nodes of the destination subcluster (lines [15][16][17][18][19][20][21][22]. Therefore, each load balancer splits its output into M substreams (line 13), according to a hash of the tuples %M (line 14).…”

“…There has been recent work on exploiting peer-to-peer networks, in particular, distributed hash tables (DHTs) for processing Continuous multi-way joins over data streams [18] [19]. Although these works exploit hash-based join algorithms, the objective (increasing the size of the sliding window with addition of peers) is different than ours (scaling out) and the assumptions regarding the network (clusters vs. WANs) are very different.…”

Fault-tolerant query processing in structured P2P-systems

“…The problem of failures during query processing in distributed data management systems has received a lot of attention [21][22][23]. Palma et al [21] has identified the problem of peer failures while processing join operations over distributed data streams.…”

“…Palma et al [21] has identified the problem of peer failures while processing join operations over distributed data streams. The approach addresses unnecessary communication and aborts the execution on peers executing subsequent operators of the query if a failure has been detected.…”