Handling Data-skew Effects in Join Operations Using MapReduce

Hassan, Mohamad Al Hajj; Bamha, Mostafa; Loulergue, Frédéric

doi:10.1016/j.procs.2014.05.014

Cited by 25 publications

(24 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The algorithm optimizes multiway join by minimizing duplicate tuples of input data and can detect whether an attribute is incorrectly contained in the keys of map side and then fix it. Hassan et al proposed a new frequency adaptive algorithm called MRFA‐Join for join operations of large‐scale data sets based on a MapReduce programming model. To solve the problem of buffering all records of internal and external relations, Blans et al proposed a semi‐join algorithm for log processing and improve the sorting and merging operations in MapReduce.…”

Section: Related Workmentioning

confidence: 99%

Handling data skew at reduce stage in Spark by ReducePartition

Guo

Huang

Tian

2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary As a typical representative of distributed computing framework, Spark has been continuously developed and popularized. It reduces the data transmission time through efficient memory‐based operations and solves the shortcomings of the traditional MapReduce computation model in iterative computation. In Spark, data skew is very prominent due to the uneven distribution of input data and the unbalanced allocation of default partitioning algorithm. When data skew occurs, the execution efficiency of the program will be reduced, especially in the reduce stage of Spark. Therefore, this paper proposes ReducePartition to solve data skew problem at reduce stage of Spark platform. First, the compute node samples the local data according to the sampling algorithm to predict the overall characteristics of data distribution. Then, to take full use of cluster resources, ReducePartition divides data into multiple partitions evenly. Next, taking into account the differences in computational capabilities among Executors, each task is assigned to Executor with the highest performance factor according to the greedy strategy. Finally, the results of the related algorithms and ReducePartition are compared by using WordCount benchmark and Sort benchmarks on heterogeneous Spark standalone cluster. The performance of the ReducePartition under different degree of data skew and different data size is analyzed. Experimental results show that the proposed algorithm can effectively reduce the impact of data skew on the total makespan of Spark big data applications, and the average total makespan is reduced by 30% to 50% while resource utilization is increased by 20%‐30% on average.

show abstract

Section: Related Workmentioning

confidence: 99%

Handling data skew at reduce stage in Spark by ReducePartition

Guo

Huang

Tian

2019

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…Regarding joins in a cloud computing environment, most of current works focus on proposing novel data skew handling techniques to improve the loadbalancing and scalability of join implementations in the presence of big data (e.g., SALA [35], SkewTune [36] and the approaches presented in [37], [38]), as opposed to the detailed implementation and evaluation of joins that is studied in this work. Moreover, the large scale data-analytics community has developed its own set of parallel processing paradigms and related join operations.…”

Section: Related Workmentioning

confidence: 99%

Design and evaluation of small–large outer joins in cloud computing environments

Cheng

Tachmazidis

Kotoulas

et al. 2017

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

show abstract

“…For a highly skewed join attribute value K, appropriate map keys are generated so that all records in each bucket associated to value K in one relation are forwarded to the same reducer holding all the corresponding buckets of other relation. This partitioning guarantees that join tasks, are generated in a manner that the input data for each join task will fit in the memory of processing node and never exceed a user defined size, even for highly skewed data [10].…”

Section: End Ifmentioning

confidence: 99%

“…Business intelligence and large-scale data analysis have been recently the object of increased research activity using MapReduce model and especially in the evaluation of complex queries involving GroupBy-Joins using hash based approach [2,10,14]. GroupBy-joins still suffer from the effect of high redistribution cost, disk I/O and task imbalance in the presence of skewed data in large scale systems.…”

Section: Introductionmentioning

confidence: 99%

“…We have recently proposed in [10], MRFA-Join algorithm, a MapReduce based algorithm for evaluating join operations on Distributed File Systems (DFS). MRFA-Join algorithm is a scalable and skew-insensitive join algorithm based on distributed histograms and on a randomised key redistribution approach while guaranteeing perfect balancing during all stages of join computation.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Scalability and Data Skew Handling in GroupBy-Joins using MapReduce Model

Hassan

Bamha

2015

Procedia Computer Science

Self Cite

View full text Add to dashboard Cite

For over a decade, MapReduce has become the leading programming model for parallel and massive processing of large volumes of data. This has been driven by the development of many frameworks such as Spark, Pig and Hive, facilitating data analysis on large-scale systems. However, these frameworks still remain vulnerable to communication costs, data skew and tasks imbalance problems. This can have a devastating effect on the performance and on the scalability of these systems, more particularly when treating GroupBy-Join queries of large datasets.In this paper, we present a new GroupBy-Join algorithm allowing to reduce communication costs considerably while avoiding data skew effects. A cost analysis of this algorithm shows that our approach is insensitive to data skew and ensures perfect balancing properties during all stages of GroupBy-Join computation even for highly skewed data. These performances have been confirmed by a series of experimentations.

show abstract

Handling Data-skew Effects in Join Operations Using MapReduce

Cited by 25 publications

References 10 publications

Handling data skew at reduce stage in Spark by ReducePartition

Handling data skew at reduce stage in Spark by ReducePartition

Design and evaluation of small–large outer joins in cloud computing environments

Towards Scalability and Data Skew Handling in GroupBy-Joins using MapReduce Model

Contact Info

Product

Resources

About