A Comparative Study of Data Skew in Hadoop

He, Majun; Li, Guozhong; Huang, Chaojie; Ye, Yufei; Tian, Wenhong

doi:10.1145/3171592.3171610

Cited by 5 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In such case, using HDFS blocks directly to estimate statistics and build models may lead to statistically incorrect or biased results. Another key issue is data skew, which describes the uneven distribution of the records leading to tasks with different execution times [67][68][69] . On computing clusters, the performance strongly depends on how evenly data are distributed among the nodes.…”

Section: Big Data Partitioning On Hadoop Clustersmentioning

confidence: 99%

A survey of data partitioning and sampling methods to support big data analysis

Mahmud

Huang

Salloum

et al. 2020

Big Data Min. Anal.

218

View full text Add to dashboard Cite

Computer clusters with the shared-nothing architecture are the major computing platforms for big data processing and analysis. In cluster computing, data partitioning and sampling are two fundamental strategies to speed up the computation of big data and increase scalability. In this paper, we present a comprehensive survey of the methods and techniques of data partitioning and sampling with respect to big data processing and analysis.We start with an overview of the mainstream big data frameworks on Hadoop clusters. The basic methods of data partitioning are then discussed including three classical horizontal partitioning schemes: range, hash, and random partitioning. Data partitioning on Hadoop clusters is also discussed with a summary of new strategies for big data partitioning, including the new Random Sample Partition (RSP) distributed model. The classical methods of data sampling are then investigated, including simple random sampling, stratified sampling, and reservoir sampling. Two common methods of big data sampling on computing clusters are also discussed: record-level sampling and blocklevel sampling. Record-level sampling is not as efficient as block-level sampling on big distributed data. On the other hand, block-level sampling on data blocks generated with the classical data partitioning methods does not necessarily produce good representative samples for approximate computing of big data. In this survey, we also summarize the prevailing strategies and related work on sampling-based approximation on Hadoop clusters. We believe that data partitioning and sampling should be considered together to build approximate cluster computing frameworks that are reliable in both the computational and statistical respects.

show abstract

Section: Big Data Partitioning On Hadoop Clustersmentioning

confidence: 99%

A survey of data partitioning and sampling methods to support big data analysis

Mahmud

Huang

Salloum

et al. 2020

Big Data Min. Anal.

218

View full text Add to dashboard Cite

show abstract

“…1. Calculate the average number of records assigned to each partition p avg according to estimated intermediate data tuples and Formula (9), and set the maximum amount of data to be allocated for each partition to p avg .…”

Section: Data Balanced Partition Algorithmmentioning

confidence: 99%

“…He. et al provided a comparative study on data skew in Hadoop in terms of architectures, main features, core algorithms, performance metrics, and evaluation methods, and summarized a few challenging problems as future research trends.…”

Section: Related Workmentioning

confidence: 99%

Handling data skew at reduce stage in Spark by ReducePartition

Guo

Huang

Tian

2019

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Summary As a typical representative of distributed computing framework, Spark has been continuously developed and popularized. It reduces the data transmission time through efficient memory‐based operations and solves the shortcomings of the traditional MapReduce computation model in iterative computation. In Spark, data skew is very prominent due to the uneven distribution of input data and the unbalanced allocation of default partitioning algorithm. When data skew occurs, the execution efficiency of the program will be reduced, especially in the reduce stage of Spark. Therefore, this paper proposes ReducePartition to solve data skew problem at reduce stage of Spark platform. First, the compute node samples the local data according to the sampling algorithm to predict the overall characteristics of data distribution. Then, to take full use of cluster resources, ReducePartition divides data into multiple partitions evenly. Next, taking into account the differences in computational capabilities among Executors, each task is assigned to Executor with the highest performance factor according to the greedy strategy. Finally, the results of the related algorithms and ReducePartition are compared by using WordCount benchmark and Sort benchmarks on heterogeneous Spark standalone cluster. The performance of the ReducePartition under different degree of data skew and different data size is analyzed. Experimental results show that the proposed algorithm can effectively reduce the impact of data skew on the total makespan of Spark big data applications, and the average total makespan is reduced by 30% to 50% while resource utilization is increased by 20%‐30% on average.

show abstract

Data balancing-based intermediate data partitioning and check point-based cache recovery in Spark environment

Cai

Luo

2021

J Supercomput

View full text Add to dashboard Cite

A Comparative Study of Data Skew in Hadoop

Cited by 5 publications

References 10 publications

A survey of data partitioning and sampling methods to support big data analysis

A survey of data partitioning and sampling methods to support big data analysis

Handling data skew at reduce stage in Spark by ReducePartition

Data balancing-based intermediate data partitioning and check point-based cache recovery in Spark environment

Contact Info

Product

Resources

About