Robust and Efficient Large-Large Table Outer Joins on Distributed Infrastructures

Abstract

The full-text may be used and/or reproduced, and given to third parties in any format or medium, without prior permission or charge, for personal research or study, educational, or not-for-prot purposes provided that:• a full bibliographic reference is made to the original source • a link is made to the metadata record in DRO • the full-text is not changed in any way The full-text must not be sold in any format or medium without the formal permission of the copyright holders.Please consult the full DRO policy for further details. Abstract. Outer joins are ubiquitous in many workloads but are sensitive to loadbalancing problems. Current approaches mitigate such problems caused by data skew by using (partial) replication. However, contemporary replication-based approaches (1) introduce overhead, since they usually result in redundant data movement, (2) are sensitive to parameter tuning and value of data skew and (3) typically require that one side is small. In this paper, we propose a novel parallel algorithm, Redistribution and Efficient Query with Counters (REQC), aimed at robustness in terms of size of join sides, variation in skew and parameter tuning. Experimental results demonstrate that our algorithm is faster, more robust and less demanding in terms of network bandwidth, compared to the state-of-the-art.

A two-tier index architecture for fast processing large RDF data over distributed memory

“…We will investigate extensions to our design through the application of methods for skew handling [15,5,6,7], index size reduction [14] and incremental sorting [12,13] which should further improve performance. Our long term goal is to develop a highly scalable distributed analysis framework for extreme-scale RDF data.…”

Investigating Distributed Approaches to Efficiently Extract Textual Evidences for Biomedical Ontologies

“…(2) Cardinality: To see how the performance changes with increasing dataset size, similar as the evaluation works on joins [13] [14], we just fix the cardinality of relation R, to 25 million, and varying the |S| from 25 million to 400 million, a number which is extreme big for the available annotation pairs. As the skew handling is beyond the scope of this work, we just keep the data uniform distributed based on their first join key a as stated previously.…”

Robust and Skew-resistant Parallel Joins in Shared-Nothing Systems

“…Compared to these, in our previous work [7,8,9], we have employed the semijoin-alike pattern with full parallelism as a new distributed geography (namely not just a simple join operation) for handling data skew and apply it for parallel inner joins and outer joins directly. In this work, we focus on the inner joins (namely joins).…”

“…We conclude our analysis with the presentation of speedup using the very popular Hash algorithm as a baseline 9 , by analyzing the performance improvement achieved for joins in each algorithm for different numbers of nodes. Figure 9 presents the speedup ratio of PRPD, PRPQ and the Query algorithm over the basic hash method with increasing number of nodes from 2 (24 cores) to 16 and for skew values 1 and 1.4 respectively.…”